Thanks for the much anticipated update! Appreciate that you provide a roadmap for the future. Hopefully there aren't too many roadblocks ahead with the development of OpenMM. The future project direction sounds very exciting :) I'll take that as an opportunity to upgrade my host by the end of the year to contribute more intensively next year!

Keep up the good work

Are there any plans to add support for AMD GPUs now that ACEMD3 supports OPENCL? https://software.acellera.com/docs/latest/acemd3/capabilities.html This would increase participation.

Good news, everyone!

Great News, Thanks!!!!

Thanks for the news, I hope the work goes well. I am looking forward to making new calculations.

Go ahead !

I've got a 3090 and 2 1080ti's waiting for some work. Looking forward to the new updates.

On the practical term, we expect to have the ACEMD application fixed for RTX30xx within few weeks, as now the developper of ACEMD is also doing the deployment on GPUGRID, making everything simpler.

Are there any plans to add support for AMD GPUs now that ACEMD3 supports OPENCL? https://software.acellera.com/docs/latest/acemd3/capabilities.html This would increase participation.

Some initial new version of ACEMD has been deployed on linux and it's working, but we are still testing.

gdf

Support for AMD cards would make good for project.
Atm. I´m running mostly Milkyway with my 6900XT and its doing 3 units in 1:50... that takes for 1080Ti about 6:00. I havent checked times to WCG because GPU units cames up so rarely, but I can imagine it to be ok in that too.

Some initial new version of ACEMD has been deployed on linux and it's working, but we are still testing.

gdf

Haven't snagged one of the new ones as yet (I was out all day), but I'll watch out for them, and try to find out where the device allocation is failing.

there seems to be a problem with the new 2.17 app. it's always trying to run on GPU0 even when BOINC assigns it to another GPU.

I had this happen on two separate hosts now. the host picked up a new task, BOINC assigns it to some other GPU (like device 6 or device 3) but the acemd process spins up on GPU0 anyway, even though it is already occupied by another BOINC process from another project. I think there's sometime off in that the boinc device assignment isnt being communicated to the app properly. this results in multiple processes running on a single GPU, and no process running on the device that BOINC assigned the GPUGRID task to.

there seems to be a problem with the new 2.17 app. it's always trying to run on GPU0 even when BOINC assigns it to another GPU.

First of all: Congratulations, both of your multi GPU systems that weren't getting work from previous app versions, seem to have the problem solved with this new one. Welcome back to the field!

Great news! Now to just have some tasks ready to send.

Good news! :-)

Got 1 WU.

Boinc shows a runtime of 6 days on my 2080ti with 60% powertarged. Hoppefully its not the real time.

Edit: 1% after 10 Minutes so probably something about 16h

Picked up a task apiece on two hosts. Some 800 tasks in progress now.

Hope this means the project is getting back to releasing steady work.

it seems that even though the cuda1121 app is available and works fine on Ampere cards, there's nothing preventing the cuda101 app from being sent to an Ampere host. these will always fail.

example: https://gpugrid.net/result.php?resultid=32643471

The project-side scheduler needs to be adjusted to not allow the cuda101 app from being sent to Ampere hosts. this can be achieved by checking the compute capability reported from the host. In addition to the cuda version checks, the cuda101 app should be limited to hosts with compute capability less than 8.0. hosts with 8.0 or greater should only get the cuda1121 app.

or simply remove the cuda101 app and require all users to update their video drivers to use the 1121 app.
____________

This is correct. I had the same exact failure with the CUDA101 app sent to my Ampere RTX 3080.

Failure was the inability to compile the CUDA kernel because it was expecting a different architecture.

https://www.gpugrid.net/result.php?resultid=32642922

Technical question: has anybody idea if the CC is available in the scheduler?

<min_nvidia_compcap>MMmm</min_nvidia_compcap>
minimum compute capability
<max_nvidia_compcap>MMmm</max_nvidia_compcap>
maximum compute capability

To expand on what Richard wrote, I’m sure it’s available. Einstein@home uses this metric in their scheduler to gatekeep some of their apps to certain generations of Nvidia GPUs. So it’s definitely information that’s provided from the host to the project via BOINC.
____________

So it’s definitely information that’s provided from the host to the project via BOINC.

Uhm... yes but I was wondering how to retrieve it in the C++ code.

Uhm... yes but I was wondering how to retrieve it in the C++ code.

Won't work for multi-gpu users that have both Turing and Ampere cards installed in a host.

I have a 2080 and 3080 together in a host.

Won't work for multi-gpu users that have both Turing and Ampere cards installed in a host.

I have a 2080 and 3080 together in a host.

Can I use it on multi-GPU systems?

In general yes, with one caveat: if you have DIFFERENT types of NVIDIA GPUs in the same PC, suspending a job in one and restarting it in the other will NOT be possible (errors on restart). Consider restricting the client to one GPU type only ("exclude_gpu", see here).

There are two different aspects to this debate:

1) What will a project server send to a mixed-GPU client?
2) Which card will the client choose to allocate a task to?

The project server will allocate work solely on the basis of Keith's 3080. BOINC has been developed, effectively, to hide his 2080 from the server.

Keith has some degree of control over the behaviour of his client. He can exclude certain applications from particular cards (using cc_config.xml), but he can't exclude particular versions of the same application - the control structure is too coarse.

He can also control certain behaviours of applications at the plan_class level (using app_config.xml), but that control structure is too fine - it doesn't contain any device-level controls.

Other projects have been able to develop general-purpose GPU applications which are at least compatible with mixed-device hosts - tasks assigned to the 'wrong' or 'changed' device at least run, even if efficiency is downgraded. If this project is unable to follow that design criterion (and I don't know why it is unable at this moment), then I think the only available solution at this time is to divide the versions into separate applications - analogous to the old short/long tasks - so that the limited range of available client options can be leveraged.

I haven’t seen enough compelling evidence to justify keeping the cuda101 app. The cuda1121 app works on all hosts and is basically the same speed.

Removing the cuda101 app would solve all problems.
____________

I agree. Simply remove the CUDA101 app and restrict sending tasks to any host that hasn't updated the drivers to the CUDA11.2 level.

What's with the new ACEMD beta version 9.17, introduced today? What are we testing?

I got a couple of really short tasks on Linux host 508381. That risks really messing up DCF.

I received 15 of the test wu's no problems on Ampere all crunched without issue just want more :)

Yes, you crunched those tasks with the CUDA1121 app on your Ampere which was always working.

A beta test should have been with the CUDA101 app on Ampere to see whether they have fixed the issue with all tasks failing

NVIDIA GeForce RTX 3080 TI
Driverversion 472.12 Game Ready Driver

e2s94_e1s550p0f1071-ADRIA_AdB_KIXCMYB_HIP-1-2-RND9846_0

195 (0xc3) EXIT_CHILD_FAILED
New version of ACEMD v2.18 (cuda101)

<core_client_version>7.16.11</core_client_version>
<![CDATA[
<message>
(unknown error) - exit code 195 (0xc3)</message>
<stderr_txt>
20:09:10 (720): wrapper (7.9.26016): starting
20:09:10 (720): wrapper: running bin/acemd3.exe (--boinc --device 0)
ACEMD failed:
Error compiling program: nvrtc: error: invalid value for --gpu-architecture (-arch)

Mine also failed on a single 3080, driver 470.63:

Application version ACEMD beta version v9.17 (cuda101)
Stderr output

<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
process exited with code 195 (0xc3, -61)</message>
<stderr_txt>
19:20:04 (445272): wrapper (7.7.26016): starting
19:20:12 (445272): wrapper (7.7.26016): starting
19:20:12 (445272): wrapper: running bin/acemd3 (--boinc --device 0)
ACEMD failed:
Error compiling program: nvrtc: error: invalid value for --gpu-architecture (-arch)

19:20:13 (445272): bin/acemd3 exited; CPU time 0.782133
19:20:13 (445272): app exit status: 0x1
19:20:13 (445272): called boinc_finish(195)

</stderr_txt>
]]>

Yes, you crunched those tasks with the CUDA1121 app on your Ampere which was always working.

A beta test should have been with the CUDA101 app on Ampere to see whether they have fixed the issue with all tasks failing

I received 15 of the test wu's no problems on Ampere all crunched without issue just want more :)

Yes, you crunched those tasks with the CUDA1121 app on your Ampere which was always working.

A beta test should have been with the CUDA101 app on Ampere to see whether they have fixed the issue with all tasks failing

They’ll never get cuda101 working on Ampere as long as they have the architecture check.

Are we out of work again? I am going to have to greylist GPUGrid again unless I see better WU flow soon.
____________

Looks like it. I was hoping the new work from the new researcher doing machine learning with pytorch was going to provide consistent work again.

But this post from Toni differs. https://www.gpugrid.net/forum_thread.php?id=5283&nowrap=true#57849

Until we fill these positions, we have little capacity to send jobs.

Project has been greylisted again for Gridcoin. 😱

Looks like it. I was hoping the new work from the new researcher doing machine learning with pytorch was going to provide consistent work again.

But this post from Toni differs. https://www.gpugrid.net/forum_thread.php?id=5283&nowrap=true#57849

Until we fill these positions, we have little capacity to send jobs.

Didn't help the WAS and ZCD scores either that the server script for generating work credits export wasn't running for 5 days either.

If project ostracism takes much longer, there is a certain risk of gradually losing its well earned prestige over years.
Usually crisis lead to changes.
I wish this time changes-to-better are being prepared.

Actually we could have a approximate date of the "restart" of the project.

I read the PhD. proposition and it's kind of an "ASAP" to start working.

But I'm quite sure there's paperwork, review of candidates etc.

So... maybe there's a deadline to all that.


@ServicEnginIC
"Usually crisis lead to changes." true. Let's hope it's for the better whatever the different crisis for now.
(makes me think of "The times they are a-changin' of Bob Dylan")
Never got the chance to tell you but nice picture as avatar.

The project won't be whitelisted again until it can consistently create work and have its credits exported for enough days to get its work available and zero credit days scores below greylist criteria.

You can verify here.https://gridcoin.ddns.net/pages/project-list.php

The fact that your stats creation script fails frequently causes it to be greylisted often along with the projects inability create consistent work does also

and what is the point of gridcoin?

Gridcoin rewards citizen scientists for their distributed computing contribution.

The Gridcoin network can reward your project's userbase to encourage participation and improve user retention.

Gridcoiners can also send GRC directly to researchers for their project through the project's wallet address or can side-stake a portion of their research awards to project scientists.

The Gridcoin protocol distributes GRC directly to scientists and those who contribute to scientific endeavors. Every time you use GRC, you voice your support for an economy based on science.

The human foot did not evolve with shoes.

can't someone pls just block this guy/bot. guy's posting all over random stuff and cluttering message boards across various projects. thx

Admin,
please, block this "administrator" (userid 573008)
he (or it) publish non sense ,and not only here !!!

marsinph that is a hassle to do for admin but you can block him in your end.

marsinph that is a hassle to do for admin

Incidentally how does gridcoin works and where is the list of project gray listed?

The website for the status of project whitelisting https://gridcoin.ddns.net/pages/project-list.php
isn't being maintained anymore. G-UK stated this to me recently when I added an issue to the code repo for the page. So you can't depend on that for true status.

But the wallet client Help/Debug window/Scraper tab does show the status of current whitelisted projects. Unlisted or greylisted projects fall off the scraper convergence tally.

Also the Main page of the wallet has the gear icon which takes you to the Researcher configuration page with the Summary and Projects tabs. The Projects tab lists all the projects and their status of listed or unlisted and if there is magnitude shown, you know that it is still listed. This updates at every Superblock.

I had a pair of WUs complete successfully on a computer with a pair of 2080 Ti's. My other GG computer with a 3080 + 3080 Ti has failed 6 times. E.g.,

If you have different devices, you can't interrupt the acemd3 calculation or you will error it out on restart. The only solution is to change your "switch between applications every xx minutes" in the Manager and pass that to the client.

Set that to a value that the longest running tasks have run on the host and add 10%. I have mine set at 2880 minutes or two days. The tasks run to completion on the card and then the card can move on to other gpu work.

To add on to what Keith said, it’s become apparent that the acemd3 app is generating some memory load which is specific to the hardware it’s running on. Look at different GPUs and you’ll see different amounts of memory used by each, and it seems to scale by total memory size. (IE, a 12GB GPU will show more memory used than a 4GB GPU). Or maybe it scales by memory configuration (bandwidth, speed) and not just total size. Or some combination.

This is why you can’t restart a task on a different device. The calculation which was setup for some specific hardware cannot continue if you change the device hardware midway through. They seem to have some logic built in to catch when hardware has changed. If you have identical GPUs, most times it will restart on a different device OK, but I’ve seen times when even restarting on an identical GPU still triggers this and it fails right away.

Best option is to never interrupt GPUGRID tasks.
____________

At the root of this problem is why does GG switch to a different device when its name did not change???

If another GPU WU switches to Running High Priority it will override that switch every 2880 minutes approach.

When we get a continuous supply of acemd3 WUs my urge to timeslice them will vanish :-)

Has nothing to do with GPUGrid or the acemd3 application. The issue is with BOINC.

BOINC does not care that you were running on any particular device. It just knows that a gpu resource just became available when a task finishes on a gpu and assigns the interrupted or checkpointed acemd3 task to the card that just became available.

If that is a different device that what the task started on, then the task errors out in the manner that Ian mentioned.

Make sure your task switching options are set to longer than the estimated task run time.

As I recall, you also run WCG OPNG tasks, which if you run that at a higher priority, or some other project at a higher priority (resource share) and your task has been running longer than the task switch time, it will fail over to the new high priority task, stopping GPUGRID, then if it restarts on a different device, you get the error. But if your task switch time is longer than run time, it shouldn’t ever switch away to high priority work until the task is complete.
____________

Has nothing to do with GPUGrid or the acemd3 application. The issue is with BOINC.

BOINC does not care that you were running on any particular device. It just knows that a gpu resource just became available when a task finishes on a gpu and assigns the interrupted or checkpointed acemd3 task to the card that just became available.

If that is a different device that what the task started on, then the task errors out in the manner that Ian mentioned.

Again, the acemd3 application would have to be completely rewritten to hook into BOINC in a manner that BOINC does not currently have.

So rewrite BOINC first to get the ability to assign individual tasks to be locked to specific hardware and then rewrite GG to use that BOINC feature.

Or if the task is detected to be NOT running on the original hardware, start the the task from zero again so that it does not error. Which is not conducive to returning work in the original 5 day deadline for slower cards.

keep task switching set to the proper value and operate with the understanding of the idiosyncrasies of different projects and this won't be an issue. for GPUGRID that means not turning off your PC or anything else that would cause the task to restart while processing GPUGRID work.

I take it a step further and have designed my GPUGRID systems with identical GPUs. not just the same model, but where possible the same exact SKU from the same manufacturer (the 7x 2080Ti system has all EVGA 2080Ti XC Ultra cards; the 7x 2080 system has all ASUS 2080Ti Turbo custom watercooled). Making the system homogeneous in this way greatly reduces the chance that a restart detects a new device as "different" in the event that a restart is unavoidable (like a power outage, or hardware issue), since they are all identical. You can take this mindset even further with battery backup to cover short power outages. My smaller 1-GPU system is on a 1500W battery backup that can keep the system up for a few minutes during short power blips, which is all I usually experience in my area. Just enough that the mains voltage drop doesn't induce a reboot, the battery just kicks on for a few seconds and it stays up.
____________

I haven't received any new projects since roughly in November and I just recently replaced the Certificate. My other 3 projects are working just fine except for this one. I have tried resetting, removing and re-adding GPUGRID but no work gets downloaded. Any ideas?

Any ideas?

ok, Thanks!