Advanced search

Message boards : News : New beta application

Author Message
Profile MJH
Project administrator
Project developer
Project scientist
Send message
Joined: 12 Nov 07
Posts: 696
Credit: 27,266,655
RAC: 0
Level
Val
Scientific publications
watwat
Message 44343 - Posted: 30 Aug 2016 | 10:52:20 UTC

Hi Crunchers,

The eagle-eyed amongst you might have noticed the recent appearance of "Python Runtime" as the latest addition to our stable of applications.

* what is it? This app deploys a Python runtime environment based on the Conda python distribution.

* why do we need it? Almost all of our application development is now done in Python. This app will give us a way to reliably deploy and execute our software on GPUGRID volunteers' computers by providing a uniform, known Python environment

* how will it be used? If all works well, we'll be able to quickly test and deploy whole new applications, including replacements for the quantum chemistry and CPU MD applications we have previously experimented with. Being able to rapidly deploy new applications will markedly improve the rate at which our research group can innovate.

For the moment the app is Linux only but we hope to roll it out to Windows once it proves reliable.

Matt

kain
Send message
Joined: 3 Sep 14
Posts: 152
Credit: 887,959,459
RAC: 1,570,517
Level
Glu
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwat
Message 44344 - Posted: 30 Aug 2016 | 12:09:40 UTC

So can we expect WUs for AMD cards?

Profile MJH
Project administrator
Project developer
Project scientist
Send message
Joined: 12 Nov 07
Posts: 696
Credit: 27,266,655
RAC: 0
Level
Val
Scientific publications
watwat
Message 44345 - Posted: 30 Aug 2016 | 14:04:38 UTC - in response to Message 44344.

So can we expect WUs for AMD cards?


It's something that this new app will help us develop, yes.

Matt

kain
Send message
Joined: 3 Sep 14
Posts: 152
Credit: 887,959,459
RAC: 1,570,517
Level
Glu
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwat
Message 44346 - Posted: 30 Aug 2016 | 14:41:51 UTC

GREAT!

Profile skgiven
Volunteer moderator
Volunteer tester
Avatar
Send message
Joined: 23 Apr 09
Posts: 3968
Credit: 1,995,359,260
RAC: 0
Level
His
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 44349 - Posted: 30 Aug 2016 | 20:40:38 UTC - in response to Message 44346.

Both Betas went to the same user. That's quite a finite testing methodology. Might take a while to get round everyone though...
____________
FAQ's

HOW TO:
- Opt out of Beta Tests
- Ask for Help

zombie67 [MM]
Avatar
Send message
Joined: 16 Jul 07
Posts: 209
Credit: 4,299,911,456
RAC: 13,596,874
Level
Arg
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 44350 - Posted: 30 Aug 2016 | 23:43:40 UTC - in response to Message 44343.

* why do we need it? Almost all of our application development is now done in Python. This app will give us a way to reliably deploy and execute our software on GPUGRID volunteers' computers by providing a uniform, known Python environment


Is there anything in particular we need to have installed on our linux machines? Is python already installed by default in linux distributions?
____________
Reno, NV
Team: SETI.USA

Profile MJH
Project administrator
Project developer
Project scientist
Send message
Joined: 12 Nov 07
Posts: 696
Credit: 27,266,655
RAC: 0
Level
Val
Scientific publications
watwat
Message 44356 - Posted: 31 Aug 2016 | 12:19:19 UTC - in response to Message 44350.


Is there anything in particular we need to have installed on our linux machines? Is python already installed by default in linux distributions?


Python distributions and packaging are what's technically known as a catastrophe. We have to deploy our own stand-alone environment to have any hope of getting a reliable base to work from.

zombie67 [MM]
Avatar
Send message
Joined: 16 Jul 07
Posts: 209
Credit: 4,299,911,456
RAC: 13,596,874
Level
Arg
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 44362 - Posted: 31 Aug 2016 | 19:49:00 UTC - in response to Message 44356.


Is there anything in particular we need to have installed on our linux machines? Is python already installed by default in linux distributions?


Python distributions and packaging are what's technically known as a catastrophe. We have to deploy our own stand-alone environment to have any hope of getting a reliable base to work from.


Ah. So this will be run in a VM? We will need vbox installed? I am trying to understand what is required for the user to install to allow this to work.
____________
Reno, NV
Team: SETI.USA

[VENETO] boboviz
Send message
Joined: 10 Sep 10
Posts: 160
Credit: 388,132
RAC: 0
Level

Scientific publications
wat
Message 44392 - Posted: 2 Sep 2016 | 9:33:45 UTC - in response to Message 44345.

So can we expect WUs for AMD cards?


It's something that this new app will help us develop, yes.

Matt


Oh, my....

[VENETO] boboviz
Send message
Joined: 10 Sep 10
Posts: 160
Credit: 388,132
RAC: 0
Level

Scientific publications
wat
Message 44393 - Posted: 2 Sep 2016 | 9:35:05 UTC - in response to Message 44362.

Ah. So this will be run in a VM? We will need vbox installed? I am trying to understand what is required for the user to install to allow this to work.


+1
Or they use something like Cosmology@home? Docker app....

kain
Send message
Joined: 3 Sep 14
Posts: 152
Credit: 887,959,459
RAC: 1,570,517
Level
Glu
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwat
Message 44396 - Posted: 2 Sep 2016 | 10:10:14 UTC - in response to Message 44362.


Ah. So this will be run in a VM? We will need vbox installed?


Oh god please no...

Jacob Klein
Send message
Joined: 11 Oct 08
Posts: 1127
Credit: 1,901,927,545
RAC: 0
Level
His
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 44398 - Posted: 2 Sep 2016 | 11:54:27 UTC

Need more details ...
Will this be deployed as a VM requiring VirtualBox?

[VENETO] boboviz
Send message
Joined: 10 Sep 10
Posts: 160
Credit: 388,132
RAC: 0
Level

Scientific publications
wat
Message 44459 - Posted: 8 Sep 2016 | 9:34:00 UTC

Continuum Analytics Anaconda
http://gpuopen.com/compute-product/continuum-analytics-anaconda-with-numba/

xdarma
Send message
Joined: 12 Mar 11
Posts: 10
Credit: 106,552,864
RAC: 0
Level
Cys
Scientific publications
watwatwatwatwatwatwat
Message 44495 - Posted: 12 Sep 2016 | 20:08:54 UTC

Interesting.
Project resumed and waiting for WU.

jlhal
Send message
Joined: 1 Mar 10
Posts: 147
Credit: 1,077,535,540
RAC: 0
Level
Met
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 44500 - Posted: 12 Sep 2016 | 21:23:11 UTC - in response to Message 44495.

Interesting.
Project resumed and waiting for WU.

Also allowed this project in my prefs.
Python 3 already previously installed (but not Anaconda distrib) because required by some apps...
Waiting...
____________
Lubuntu 16.04.1 LTS x64

Stefan
Project administrator
Project developer
Project tester
Project scientist
Send message
Joined: 5 Mar 13
Posts: 348
Credit: 0
RAC: 0
Level

Scientific publications
wat
Message 44558 - Posted: 20 Sep 2016 | 8:38:52 UTC

AFAIK you don't need anything installed. Conda ships it's own python so that we can be consistent and not use the one the user/OS has installed which might be different versions / broken etc.

[VENETO] boboviz
Send message
Joined: 10 Sep 10
Posts: 160
Credit: 388,132
RAC: 0
Level

Scientific publications
wat
Message 44623 - Posted: 4 Oct 2016 | 17:41:18 UTC

0 wu of this beta app.
Why publish this news without work to submit?

ExtraTerrestrial Apes
Volunteer moderator
Volunteer tester
Avatar
Send message
Joined: 17 Aug 08
Posts: 2705
Credit: 1,311,122,549
RAC: 0
Level
Met
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 44624 - Posted: 4 Oct 2016 | 18:02:31 UTC - in response to Message 44623.

Because there were some WUs and test with them have been done. Apparently no further tests are needed at the moment, otherwise there'd be new WUs.

MrS
____________
Scanning for our furry friends since Jan 2002

[VENETO] boboviz
Send message
Joined: 10 Sep 10
Posts: 160
Credit: 388,132
RAC: 0
Level

Scientific publications
wat
Message 44666 - Posted: 12 Oct 2016 | 14:34:52 UTC - in response to Message 44624.

Because there were some WUs and test with them have been done.


As usual, it seems to me a "paper launch".
How many Android wus have you seen?
How many OpenCl?
Quantum Chemistry?
Anaconda??

Revrnd
Send message
Joined: 21 Mar 16
Posts: 6
Credit: 76,105,375
RAC: 0
Level
Thr
Scientific publications
watwat
Message 44668 - Posted: 13 Oct 2016 | 2:45:55 UTC

Hi, long time browser, first time poster. Love your work guys.

Just have one niggling question...

Way back late July [url]http://gpugrid.net/forum_thread.php?id=4341&nowrap=true#44031[/url] there was word of a new app for Pascal to take advantage of the new GTX 10x0 GPUs once Cuda 8 was released.

Afaik Cuda 8.0 came out September 28, 2016 (14 days ago) but I can't seem to find anything clear to say that the original app tested with Cuda 8.0 RC is going to be rolled out, or if a new app is being developed to replace it altogether. Given we're mid October and the year is rolling to a close soon, it'd be handy to know what's happening or planned.

It'd be great if someone could have a sticky - Moderator only GPUGRID App Status thread that lets everyone know what's current version, what's in testing, blockers (from stopping rollout) etc. as this would greatly assist in diseminating info out to all those who are treading water with Pascal GPUs (I've 2x Titan X Pascal GPUs myself sitting idle) waiting for a suitable Pascal compatible app release.

Other than that, please keep up the good work.

Profile Beyond
Avatar
Send message
Joined: 23 Nov 08
Posts: 1112
Credit: 6,162,416,256
RAC: 0
Level
Tyr
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 44669 - Posted: 13 Oct 2016 | 14:15:20 UTC - in response to Message 44668.
Last modified: 13 Oct 2016 | 14:16:15 UTC

Way back late July there was word of a new app for Pascal to take advantage of the new GTX 10x0 GPUs once Cuda 8 was released.

Afaik Cuda 8.0 came out September 28, 2016 (14 days ago) but I can't seem to find anything clear to say that the original app tested with Cuda 8.0 RC is going to be rolled out, or if a new app is being developed to replace it altogether. Given we're mid October and the year is rolling to a close soon, it'd be handy to know what's happening or planned.

It'd be great if someone could have a sticky - Moderator only GPUGRID App Status thread that lets everyone know what's current version, what's in testing, blockers (from stopping rollout) etc. as this would greatly assist in diseminating info out to all those who are treading water with Pascal GPUs (I've 2x Titan X Pascal GPUs myself sitting idle) waiting for a suitable Pascal compatible app release.

Here's the Admin and Moderator messages stating that they're waiting for NVidia and CUDA v8.5:

https://www.gpugrid.net/forum_thread.php?id=4341&nowrap=true#44342

http://www.gpugrid.net/forum_thread.php?id=4341&nowrap=true#44652

Seriously though the project can't even provide enough work to keep the current GPUs busy...

Profile Retvari Zoltan
Avatar
Send message
Joined: 20 Jan 09
Posts: 2356
Credit: 16,399,150,201
RAC: 4,082,544
Level
Trp
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 44671 - Posted: 13 Oct 2016 | 15:43:16 UTC - in response to Message 44669.

Seriously though the project can't even provide enough work to keep the current GPUs busy...

That's not true. There was no significant workunit shortage in the past months. However there was just enough to feed the active hosts.
Ironically, since your post Stefan announced the CASP batch, which will provide plenty of short and long workunits as well.

Profile Beyond
Avatar
Send message
Joined: 23 Nov 08
Posts: 1112
Credit: 6,162,416,256
RAC: 0
Level
Tyr
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 44684 - Posted: 13 Oct 2016 | 23:35:54 UTC - in response to Message 44671.

Seriously though the project can't even provide enough work to keep the current GPUs busy...

That's not true. There was no significant workunit shortage in the past months. However there was just enough to feed the active hosts.
Ironically, since your post Stefan announced the CASP batch, which will provide plenty of short and long workunits as well.

Actually there was. It's good to have the CASP WUs. I've had 3 so far. On a modest GTX 670, the short 1ns took only 7 minutes. The longest type (50ns) took 6 hours.

Profile Logan Carr
Send message
Joined: 12 Aug 15
Posts: 240
Credit: 64,069,811
RAC: 0
Level
Thr
Scientific publications
watwatwatwat
Message 44692 - Posted: 14 Oct 2016 | 2:13:56 UTC - in response to Message 44684.
Last modified: 14 Oct 2016 | 2:14:54 UTC

7 minutes on a gtx 670.. Wow that's pretty impressive! How long do the long tasks take you on the gtx 670?

By long tasks I mean long runs.
____________
Cruncher/Learner in progress.

Revrnd
Send message
Joined: 21 Mar 16
Posts: 6
Credit: 76,105,375
RAC: 0
Level
Thr
Scientific publications
watwat
Message 44696 - Posted: 14 Oct 2016 | 9:57:15 UTC - in response to Message 44669.
Last modified: 14 Oct 2016 | 9:59:17 UTC

I'm having a hard time trying to understand. Unfortunately programming isn't in my CV, I simply work in Freight and Distribution.

I thought with the release of Cuda 8, that work previously carried out on a pascal compliant version would hit production and that through the evaluation of Cuda 8, Pascal and the Titan X (P) / GTX 1080 / 1070 would lead to ease of programming for the lower end (1060 / 1050) when they came out.

Given there's no news whatsoever on Cuda 8.5 (Beta / RC / Release), what is it that's in Cuda 8.5 that's not in Cuda 8? Is it because of Visual Studio / Cuda bugs? I'm assuming there's still massive underlying issues in the code that don't allow a Pascal GRID App to be released / tested.

Forgive my ignorance, just trying to get a feel for things as a whole.

Just feel kinda helpless having 22TFLOPS of GPU grunt sitting here idle (as I'm sure many others do too). I was hoping to upgrade my setup to incorporate several externally linked GTX1080 / Titan GPUs to assist with this project (and also double as an Iray rendering server for me) definitely going ahead with that plan if we can get a steady flow of Pascal WUs happening at some stage.

Profile Beyond
Avatar
Send message
Joined: 23 Nov 08
Posts: 1112
Credit: 6,162,416,256
RAC: 0
Level
Tyr
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 44699 - Posted: 14 Oct 2016 | 13:19:28 UTC - in response to Message 44692.

7 minutes on a gtx 670.. Wow that's pretty impressive! How long do the long tasks take you on the gtx 670?

By long tasks I mean long runs.

SDOERR_CASP times:
GTX 670: 1ns about 7 minutes, 5ns about 35 minutes, 20ns about 2.5 hours, 50 ns about 6 hours.
GTX 750Ti: 1ns about 9.5 minutes, 5ns about 47 minutes, 20ns about 3 hours, 50ns about 8.5 hours.

3de64piB5uZAS6SUNt1GFDU9d...
Avatar
Send message
Joined: 20 Apr 15
Posts: 285
Credit: 1,102,216,607
RAC: 0
Level
Met
Scientific publications
watwatwatwatwatwat
Message 44700 - Posted: 14 Oct 2016 | 15:52:20 UTC - in response to Message 44696.

Given there's no news whatsoever on Cuda 8.5 (Beta / RC / Release), what is it that's in Cuda 8.5 that's not in Cuda 8? Is it because of Visual Studio / Cuda bugs? I'm assuming there's still massive underlying issues in the code that don't allow a Pascal GRID App to be released / tested.


Apparently one has to join the Accelerated Computing Developer Program of Nvidia to get access to the next CUDA Toolkit version.
https://developer.nvidia.com/accelerated-computing-developer

That is why we don't find any information in the web so far. Upcoming specs, features and hotfixes of Version 8.5 are provided to registered members only. That would be my guess.

Just feel kinda helpless having 22TFLOPS of GPU grunt sitting here idle (as I'm sure many others do too). I was hoping to upgrade my setup to incorporate several externally linked GTX1080 / Titan GPUs to assist with this project (and also double as an Iray rendering server for me) definitely going ahead with that plan if we can get a steady flow of Pascal WUs happening at some stage.


+1 Same here. One 1070 and one 1080 sitting idle. But it can't be helped.
____________
I would love to see HCF1 protein folding and interaction simulations to help my little boy... someday.

ExtraTerrestrial Apes
Volunteer moderator
Volunteer tester
Avatar
Send message
Joined: 17 Aug 08
Posts: 2705
Credit: 1,311,122,549
RAC: 0
Level
Met
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 44703 - Posted: 14 Oct 2016 | 21:07:01 UTC - in response to Message 44696.

Just feel kinda helpless having 22TFLOPS of GPU grunt sitting here idle (as I'm sure many others do too).

Unfortunately I can't answer any of your questions, but would be interested in the answers too. My GTX1060 was doing POEM, which I consider a good substitute for GPU-Grid. However, they just quit because they've calculated everything they needed BOINC for by now.. so I wnet to Folding@Home. It doesn't count towards BOINc credits, but it works very well and is (IMO) good science, which might appeal to anyone interested in GPU-Grid too.

BTW: this thread is about the Python Runtime. The "GPU-Grid on Pascal" thread is currently right next to it in the news forum.. please continue the discussion there.

MrS
____________
Scanning for our furry friends since Jan 2002

Revrnd
Send message
Joined: 21 Mar 16
Posts: 6
Credit: 76,105,375
RAC: 0
Level
Thr
Scientific publications
watwat
Message 44709 - Posted: 15 Oct 2016 | 11:31:12 UTC - in response to Message 44703.

Would love to continue there, but the thread is locked......

Sorry to steer the conversation away from it's subject.

ExtraTerrestrial Apes
Volunteer moderator
Volunteer tester
Avatar
Send message
Joined: 17 Aug 08
Posts: 2705
Credit: 1,311,122,549
RAC: 0
Level
Met
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 44732 - Posted: 16 Oct 2016 | 20:12:19 UTC - in response to Message 44709.

Unlocked.

MrS
____________
Scanning for our furry friends since Jan 2002

xdarma
Send message
Joined: 12 Mar 11
Posts: 10
Credit: 106,552,864
RAC: 0
Level
Cys
Scientific publications
watwatwatwatwatwatwat
Message 45585 - Posted: 4 Dec 2016 | 10:32:39 UTC

Three months are gone and I never saw a WU.
Any news?

Profile Retvari Zoltan
Avatar
Send message
Joined: 20 Jan 09
Posts: 2356
Credit: 16,399,150,201
RAC: 4,082,544
Level
Trp
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 45586 - Posted: 4 Dec 2016 | 11:03:34 UTC - in response to Message 45585.

Three months are gone and I never saw a WU.
Any news?
This project is CUDA based, so you can use NVidia cards here.
There is an AMD app, but the work for these cards is scarce.

PappaLitto
Send message
Joined: 21 Mar 16
Posts: 511
Credit: 4,672,242,755
RAC: 0
Level
Arg
Scientific publications
watwatwatwatwatwatwatwat
Message 45590 - Posted: 4 Dec 2016 | 18:13:15 UTC - in response to Message 45585.

Three months are gone and I never saw a WU.
Any news?


All of your 100m credit is from the 680s which are nvidia cards. The AMD portion of this project very limited

xdarma
Send message
Joined: 12 Mar 11
Posts: 10
Credit: 106,552,864
RAC: 0
Level
Cys
Scientific publications
watwatwatwatwatwatwat
Message 45591 - Posted: 4 Dec 2016 | 21:12:37 UTC

Hello,
I'm aware that gpugrid runs on CUDA. And I know that there's no WU for AMD cards.
But, this thread is about a cpu application that *maybe* can evolve in a OpenCL gpu version.

The application is linux-only and I no longer use nvidia cards: so, why not try to help with the cpu of my linux box?

Cheers.

xdarma
Send message
Joined: 12 Mar 11
Posts: 10
Credit: 106,552,864
RAC: 0
Level
Cys
Scientific publications
watwatwatwatwatwatwat
Message 46797 - Posted: 31 Mar 2017 | 7:13:53 UTC

After 7 months without any WU, my processor is tired of waiting.
Project suspended, see you.

PappaLitto
Send message
Joined: 21 Mar 16
Posts: 511
Credit: 4,672,242,755
RAC: 0
Level
Arg
Scientific publications
watwatwatwatwatwatwatwat
Message 46801 - Posted: 31 Mar 2017 | 13:19:45 UTC

I also have some AMD cards that I would love to throw on this project.

Tom Miller
Send message
Joined: 21 Nov 14
Posts: 5
Credit: 1,081,640,766
RAC: 0
Level
Met
Scientific publications
watwatwatwatwatwatwat
Message 46807 - Posted: 1 Apr 2017 | 4:22:20 UTC

And once again...........The project is wasting my time and money. 100% failure rate is not acceptable.

Just wanted to start the weekend early?
____________

Profile Retvari Zoltan
Avatar
Send message
Joined: 20 Jan 09
Posts: 2356
Credit: 16,399,150,201
RAC: 4,082,544
Level
Trp
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 46810 - Posted: 1 Apr 2017 | 17:03:46 UTC - in response to Message 46807.

And once again...........The project is wasting my time and money. 100% failure rate is not acceptable.
I have negligible failure rate, and I think others have same experiences, in the other case this forum would be flooded by complaints like yours (just as it happened earlier). It is easier to help you if you specify your failures more precisely (especially when you have multiple hosts). Perhaps while you are doing it you could even find the clue yourself.
Now I've checked your tasks on your hosts.
I think you are referring to your host #189416 (i7-970, 3x GTX 770). All of the failed tasks come from GPU 1 (the middle card). This card reaches 80°C (176°F), but there are tasks which fail on that card even at 74°C (165°F). It could be that this card has some kind of failure. I would separate this card, and test it with different tools like furmark. I would also check the all power connectors (PCIe, CPU, 24-pin MB) for burn marks in this PC. The topmost card in this host reaches 87°C (189°F), which is too high.
Your GTX 780Ti's are also quite hot (and have some failures) in your other hosts.
If you see
# The simulation has become unstable. Terminating to avoid lock-up (1)
messages in the stderr output of your tasks after a couple minutes of crunching, these are most probably the sign of a too hot, too much overclocked, or other way failing card (stripes or other artifacts on the screen); in some cases this is a sign of a failing PSU.
If you see this message in the output of a workunit after 5000 steps (= it fails immediately or in a very short time after start), this is a misconfigured workunit, but in this case not much of your resources are wasted.

[VENETO] boboviz
Send message
Joined: 10 Sep 10
Posts: 160
Credit: 388,132
RAC: 0
Level

Scientific publications
wat
Message 46827 - Posted: 6 Apr 2017 | 8:01:48 UTC - in response to Message 46797.

After 7 months without any WU, my processor is tired of waiting.
Project suspended, see you.


I hate being right all the time...

Post to thread

Message boards : News : New beta application

//