threadripper PCIe

Hey, you reverted my edit about the PCIe lanes comment. Maybe my wording was not the best, but unfortunately that wasn't a bogus change. Threadripper has 64 PCIe lanes but some of it is a bit misleading due to their marketing I guess. 4x of the lanes are permanently reserved for the chipset, I think that's mostly understood (and is part of the reason we do not include them at all the PCIe lanes count). Unfortunately the other 60 lanes bifurcation configuration doesn't allow them to be used any way you want.

We have a diagram of how the Zeppelin can be configured. In particular, you can only have x48 lanes dedicated to the GPU. Those lanes come from 2 * [dies] with each die having x16 + x8 configuration dedicated to graphics/acceleration for a total of 2 * (x16+x8) = 48 lanes. Each die also delivers 2 * [dies] = 2 * (x4+x4) = x16 additional lanes. AMD steals x4 permanently for the chipset, so you're left with just x12 (x4+x4+x4) for I/O. When I last talked with AMD I was told those can actually not be combined with the other 48 lanes for high-bandwidth 60 lanes, they are effectively limited to SATA/M.2. I can try to contact AMD again to get more clarification about this. --David (talk) 14:38, 4 September 2017 (EDT)

David, AMD's slides also says the same thing about the 48 dedicated PCIe lanes. You get x16+x8+x16+x8 (x48) for GPU, 3x4 for NVMe (x12), and x4 for the chipset for a total of 64 lanes. I think in theory you might be able to get away with 3x16+1x8 for x56 but maybe there are other hurdles in the way of doing this. --At32Hz (talk) 15:28, 4 September 2017 (EDT)

I mean, on the technical level those 12 lanes should be for whatever purpose the MB motherboard manufacturers want to use it. But AMD specifically designated 48 lanes to be dedicated for multiple GPUs. That was the intention behind the edit I've made. The other 12 lanes are designated for NVMe in all of the AMD ref designs and spec guides that were given out to partners. I'm not aware of any MB that doesn't follow this or uses x8 more lanes for the GPU beyond the existing 48. From a design point of view if you allocate all 56 lanes to the GPU for a "deep learning beast", you're going to starve it with just x4 NVMe + whatever the chipset delivers (which is less than x4). So you're not really gaining anything by doing this. I also doubt such workload can consistently saturate 48 lanes anyway. --David (talk) 17:52, 4 September 2017 (EDT)

I'm ok with a line like, "PCIe lane configuration is always x16+x16+x8+x8+x4+x4+x4 with the remaining x4 for the chipset." Not everyone uses PCIe only for graphics, though, so I do not like describing 48 of the lanes as dedicated to graphics. :-) Cem (talk) 12:02, 5 September 2017 (EDT)