For GPUGrid, top cards tend to be double slot width, so well spaced out slots are desirable. A motherboard with 7 single spaced slots can support no more double spaced GPU's than a board with 4 double spaced slots!

Express lane count is also very important; the more lanes you have the more cards it can 'fully' accommodate. We know that performance drops when fewer PCIE lanes are available to the GPU. Typically the more slots used, the lower the PCIE lane availability. Often boards move from one PCIE x16 lane to two PCIE x8 lanes and then to one PCIE x8, one or two PCIE x4 slots and the fourth slot is often PCIE x1. The performance of a high end GPU in a PCIE x4 slot is significantly reduced, and would be terrible in a PCIE x1 slot. I found that even a low end GPU's performance suffered significantly in a PCIE X1 slot.

Until recently the best Motherboards have been LGA 1366, with boards for AMD CPU's being a cheaper alternative. Even now i7-980X (and similar 32nm CPU) systems fair slightly better than the i7-3960X for CPU crunching (clock for clock). So until now the only benefit of these overly expensive systems is the reduction in power, and knowing that they were somewhat future-proofed...

With AMD's latest GPU being fully PCIE3.0 compliant this has changed the picture somewhat. While we cannot use these here (yet), they offer potential. Within a few months (Feb to Apr) NVidia's GeForce 600 series will start turning up, and when they do the high end GPU's in this range will all support PCIE3.0.

Note that a PCIE3.0 X8 slot hosting a fully PCIE3.0 compliant GPU is as fast as a PCIE2.0 X16 slot. PCIE3.0 is also backward compatible with PCIE2, so existing GPU's work. IIRC 2011 boards only have 42 PCIE channels, but that is enough for one GTX 600 @ PCIE3.0 X16 and 3 GTX 600's running at PCIE3.0 X8.

I have no intention to use numerous GPU's in a single board, so I bought and presently use an i7-2600 in an MSI motherboard that has two PCIE3.0 slots. Obviously not an option if you want more than 2 GPU's. If and when both are used they drop to PCIE3.0 X8 (but again, as good as two PCIE2.0 X16).
The present card in that system is not benefiting from PCIE3.0 X16, unless it's 18.5% faster due to the overhead reduction (new PCIE3.0 algorithm) - I was just future proofing, and at the time I didn't want to spend the cash speculatively on an immature 2011 based system.

So, for future-proofing, and slightly lower running costs, you could opt for a 2011 plenum board (if you have the cash to build such a rig). If not then a less expensive 1366 based rig, with higher running costs, and if 1 or 2 high end GPU's is all you want then something based on the SB LGA1155's would do.

Anyone thinking about the forthcoming GF600 series should consider a PCIE3.0 board, and if they want an LGA2011 they should check the PCIE3.0 slot specifications. The better boards seem to be one PCIE3 X16 + three PCIE3.0 X8.

Personally, I would not consider LGA2011 until Intel's 22nm cores turn up and fall significantly in price. $1000 for a system without any GPU's is too much spent on the wrong things.

The 'Sapphire Pure Black' for AMD CPU's is only PCIE2. Not sure if there even are any PCIE3 motherboards for AMD CPU's yet?

The Asus P6T7 can only facilitate three x16 GPU's or six x8 GPU's on PCIE2.0 channels and it's single slot width, so you could only have 4 big GPU's.
____________
FAQ's

HOW TO:
- Opt out of Beta Tests
- Ask for Help

That is an interesting question which has to be answered directly with another question: how extreme do you want to get? A couple points to consider:

- running costs are going to be very high
- it's going to be very loud
- forget a case, if possible at all, it's not helping cooling
- with water cooling you can use boards with single slot spacing
- there are flexible PCIe riser cables, which let you use boards with single slot spacing and GPUs with their default coolers
- you can use GTX590 to alleviate the "slot pressure"
- using 7+ GPUs (double GPU cards count double here) may require a custom bios from the manufacturer
- BOINC may get troubles recognizing many GPUs
- you may need 2 PSUs, probably rather 80+ Platinum than Gold

So you see, there's no straight answer to "what's possible".. except "a lot" :D

MrS
____________
Scanning for our furry friends since Jan 2002

Anything over ~$2000 deserves the bespoke touch.

____________
FAQ's

HOW TO:
- Opt out of Beta Tests
- Ask for Help

You can get full PCIE raisers,


You could also use every other slots on the board directly (so long as you reinforce the back of the motherboard).
____________
FAQ's

HOW TO:
- Opt out of Beta Tests
- Ask for Help

I plan 2 GTX590

The obvious gotcha would be length - you can get several different lengths.

Try a search for 'PCIE riser', rather than PCIE raiser (uk en).
____________
FAQ's

HOW TO:
- Opt out of Beta Tests
- Ask for Help

That board seems like a solid choice. However, it's nothing special. The PCIe lanes come off the CPU, which means if you use 2 GPUs it's going to be 2 x 8x, is it is with all socket 1155 boards. And the current Sandy Bridge CPUs don't support PCIe 3, as far as I know. That's only going to be possible with Ivy Bridge (soon to come).

However, the bandwidth will be fine for GPU-Grid :)

MrS
____________
Scanning for our furry friends since Jan 2002

The choice of GPU(s) is the most important consideration.
While CPU performance is significant for GPUGrid, the more GPU's you have the more the motherboard becomes important. There are many architectures to chose from today and performance does not change much across several (i7-9xx, i7-8xx, i7-2000, i7-3900). AMD have a sizable range too, but performance is generally lower. System RAM speeds and amount tend not to make much of a difference for GPUGrid, so don't overly invest there. Ditto for the HDD.

The GPU (CC2.0/2.1), CPU and PSU are key to power and efficiency.
The SB processors offer excellent performance for their power consumption (stock ~65W crunching flat out). A mildly overclocked i7-9XX (to match the SB performance) is likely to use ~130W, which might push your PSU requirements up a notch. On the other hand a good 1366 board could support more than two GPU's at reasonable PCIE levels, and also offers triple channel RAM and more CPU threads.
In terms of crunching, it's been repeatedly demonstrated that the performance of the i7 3960X is no better than an i7-980. So at present the only benefits of these overly expensive systems is reduced power requirements and support for PCIE3, which is supposedly limited to a few recent boards, and as yet not for any NVidia GPU...

PCIE 3 X8 is as fast as PCIE 2 X16 if you can use it, but PCIE3 won't be usable at GPUGrid until a Kepler turns up, or we start using AMD's most recent GPU's and then only on the expensive LGA2011 systems.

I bought a similar board as Damaraland (Z68A-G45), partially because I like MSI/dislike others, but mostly because at the time MSI were the only company locally selling a 'potentially' PCIE3 future proofed LGA1155 mother board. I never expected to see PCIE3 performances with my GTX470; the impossible to obtain (theoretical) performance increase gained by moving from PCIE2 X16 to PCIE3 X16 would make for very little improvement in task runtimes anyway, probably ~1%. Even on a GTX580 it would be no more than 5%. The benefit would however be seen in supporting a much faster GPU.

It never even crossed my mind that the CPU would need to be PCIE3 compatible! I presumed PCIE3 compatibility lay with the motherboard and GPU. So it looks like PCIE3 motherboards are being sold when there are no existing CPU's that fit that board to make it PCIE3 capable. Asus and Gigabyte are at this too, however there is hope if you have a PCIE3 SB board; there will be a 22nm LGA1155 CPU's that will be PCIE3 compatible. If the next generation of GPU is say 60% faster than a GTX580 then using one PCIE3 x16 slot could be ~8% faster than PCIE2 x16. AnandTech demonstrated that the compute capability of an HD7970 is 9% higher using PCIE3 than when using PCIE2.

Anyway PCIE3 is fully backward compatible so I lost nothing, and later this year I will have more options if I decide to upgrade my GPU, even if it means a new 22nm 1155 CPU too.
____________
FAQ's

HOW TO:
- Opt out of Beta Tests
- Ask for Help

AnandTech demonstrated that the compute capability of an HD7970 is 9% higher using PCIE3 than when using PCIE2.

When crunching at stock the voltage on my i7-2600 is 1.18 and my i7-2600K is 1.05V. The i7-2600K uses 65W crunching; 30W less than the TDP, or just under 70% of the TDP. If IB with it's 77W TDP performs similarly then it's likely to use around 54W when crunching. The clock of the top IB is 'disappointingly' the same as the SB, 3.5GHz, and there are no more cores, but if it performs as expected, around 15% better clock for clock, then that would be the equivalent of a SB at 4GHz and only using 54W.

In terms of performance per Watt that's about 38% better than SB and at the same price. Tick-tock!?! Yes, but - for a crunching system with a top GPU the power saving would probably be <5% of the entire systems draw. Suddenly that 15% CPU boost doesn't look that special, and nor should it, GPU's are where the heavy work is done. To me Intel doesn't want 1155 to do too well. Intel began with dual channel RAM, rather than triple and 5years after introducing a quad core CPU we are still stuck with 4 cores for desktops. Why not a 4GHz IB, or at least 3.8GHz (without turbo)? It would have been inside 95W. Even a 3GHz 6 core CPU would have been ~95W.

So for GPU crunching the only real benefit of IB on LGA1155 is just PCIE3 support, assuming you have a PCIE3 capable motherboard. Even then this would only noticeably benefit those who purchase a high end PCIE3 capable GPU.
Anyway IB is due on 8th Apr (2months), and NVidia will probably release their big Kepler's around that time.

I would not want to be buying a system for crunching right now; to many not so great options, but if you must there are still choices. Either get a cheap GPU such as a GTX470 or a good GPU like the GTX570. A good GPU will still be a good GPU in 3months and in 6 to 12months you would still get a reasonable return for a SB and a GTX570.

The system choices are:
Something AMD to keep the cost down, an i7-800 or i7-900 based system. All of these have reduced/no upgrades paths for CPU and no PCIE3 capabilities. A bit short sighted and heavy on the running costs, but possibly lighter on the up-front costs, especially with an AMD system.

A SB with a PCIE3 capable motherboard. OK if you will be happy to upgrade to IB and want to get a replacement GPU (or perhaps two, but no more) in the future.

Alternatively you could wait until next week and content yourself with a $285 LGA2011 Core i7 3820 for a year. This will allow you to upgrade/add a PCIE3 GPU whenever they turn up at a reasonable price, say six to nine months, and without having to upgrade the CPU. Should you want to upgrade the CPU then there are existing 3960X and 3930K 12thread 32nm processors, a potential 3980X, probably within the next 6months, and a 22nm IB will come along in the form of Ivy Bridge-E some time in the more distant future (Q4).
____________
FAQ's

HOW TO:
- Opt out of Beta Tests
- Ask for Help

Sure, for GPU-Grid the CPU performance doesn't matter. Any Quad would be too much ehre to serve 2 GPUs. However, in BOINC land we sometimes run non-GPU stuff on our CPUs, which is very IB would be an excellent choice.

i7 800: outdated and you don't save much by going for S1156 compared to S1155.
i7 900: even more outdated and 30 W higher idle power consumption of the system. Bad choice (in the context of this thread).

@SK: expect ~2% more CPU performance per clock from IB compared to SB. 15% would be massive, I doubt even Haswell will be able to pull this off (except using new instructions).

Regarding higher clocks: Intel doesn't think it's neccessary right now. And they're probably right about this..

Regarding more cores: Intel is happy to give these to you.. in a fancy socket 2011 dress with 4 memory channels. Going triple channel on the mainstream platform wouldn't have been cost effective.

MrS
____________
Scanning for our furry friends since Jan 2002

Idle power of a cruncher PC? This argument made me LOL. :D

The i7-800's can only support 8thread CPU's (plenty for most), whereas the i7-980 and similar are 32nm 12thread processors. So for those of us that crunch CPU projects,
140W/8=17.5 and 170/12=14.16 W/thread.
So if you want more threads it's 1366 or 2011. 1366 is cheaper but doesn't support PCIE3. 2011 is more expensive to purchase but cheaper to run and 'some' boards support PCIE3. Lets hope April's arrivals are PCIE3 competent.
____________
FAQ's

HOW TO:
- Opt out of Beta Tests
- Ask for Help

Compute centers usually calculate with a 1:1 ratio between generated heat and power needed to cool it down. So I agree: running PCs "just for fun" with an AC is not the best idea in the world.. although I understand it's kind of normal in some southern US states.

Running in winter: in Europe and colder regions it's normal to use much more (money-)efficient heating than electricity. However, as far as I understand, it's rather normal in (maybe again southern?) US states to heat electricity. In this case it's a 1:1 exchange, so "basically free".

MrS
____________
Scanning for our furry friends since Jan 2002

Been crunching a while and had 2 different systems fail due to heat so be careful if you try to cram too many hot things is a box.

Had a 6 core 1055t with 3x 9800GT cards crunching and the mother board caught on fire!

Had 2x ATI HD 5770 cards crunching and the hot one (the top one that had restricted air flow) gave up the ghost. Separate those cards if you want them to last.

Or use something like this:
http://www.netstor.com.tw/_03/03_02.php?ODI=
There are several different sizes available.

I'm having a little trouble with the math but assuming I'm reading the specs correctly that Turbo Box will be a bottleneck. It looks like the Turbo Box runs from a 4x link that must be shared between video cards. The stated top speed is 20Gb/s (note the lower case "b" indicating bits).

Would not recommended that to crunch with a top GPU, and I would not go with a laptop based system anyway. It would cost less to build a desktop box than go with that thing.
____________
FAQ's

HOW TO:
- Opt out of Beta Tests
- Ask for Help

I'm having a little trouble with the math but assuming I'm reading the specs correctly that Turbo Box will be a bottleneck. It looks like the Turbo Box runs from a 4x link that must be shared between video cards. The stated top speed is 20 Gb/s (note the lower case "b" indicating bits).