HOW TO: Install PCIe cards in your 2019 Mac Pro

The 2019 Mac Pro allows to connect PCIe cards. Apple gives some recommendations on how to install and configure them. But below are some more recommendations.

How to open the "Expansion Slot Utility"?

As explained in the article, to verify that your card are properly installed, you should open the Expansion Slot Utility. To do that, go to the Apple menu  > About this Mac, and then click on the "Expansion Slot Utility" button:

This should open a window where you can see at the bottom if you are overusing the PCIe pools or not:

In the example above, you can see that each pool is used at 100%, so it is not possible to add any more PCIe card (even though there are available slots), or there may be bandwidth issue to access the card. 

By default, macOS will configure automatically the PCIe Pools depending one the cards that you connect, so we recommend to leave this to automatic. But for some reasons, you may want to configure this manually (for example if you want to be sure that one card such as the afterburner will always have its required bandwidth).

WARNING! Thunderbolt devices will also use some bandwidth of pool B. See below for more details.

What are the PCIe specs of the Mac Pro?

The 2019 Mac Pros use CPU out of Intel's Xeon W chip family:

• 8-Core 3.5GHz Intel Xeon W-3223
• 12-Core 3.3GHz Intel Xeon W-3235
• 16-Core 3.2GHz Intel Xeon W-3245
• 24-Core 2.7GHz Intel Xeon W-3265M
• 28-Core 2.5GHz Intel Xeon W-3275M

If you look at the specs of these processors, you will se that there are in total 64 lanes of PCIe, which are used as follows:

• Slot 1 & 3 each use 16 lanes of PCIe 3.0, for a total of 32 lanes
• All the other PCIe slots (and Thunderbolt connections) share the remaining 32 lanes, by using 2 allocation pools of 16 lanes each. By default, macOS will automatically allocate the pools. But you can manually configure this.

So in very short:

• Slot 1: 16 lanes
• Slot 3: 16 lanes
• Pool A: 16 lanes
• Pool B: 16 lanes

 

About slots 1 to 4

Slots 1 to 4 are quite specific as these can host the 2 MPX modules. When the 2 MPX modules are connected, you won't be able to use any of these slots. And they are quite specific:

• As explained above, slots 1 & 3 each have 16 lanes of PCIe 3.0 directly linked to the CPU. Indeed, as these slots are meant to host the graphics cards, you don't want them to have to share the bandwidth with other devices.
• Slots 2 and 4 share 8 lanes of PCIe 3.0 with the MPX connector. These 2 slots are used to provide PCIe bandwidth to the Thunderbolt ports that are available on the card. If you connect a "simple" graphics card only to slot 1 or 3, slots 2 and 4 should be available to provide 8 lanes of PCIe 3.0, that will be part of the 2 pools.

What are the limits of the PCIe Pools?

As said above, there are 2 pools of 16 lanes each. So each pool can host either:

• 1 x 16 lane PCIe 3.0 card
• 2 x 8 lane PCIe 3.0 cards
• 4 x 4 lane PCIe 3.0 cards

Note that if you are connecting a PCIe 2.0 card, the speed is half the PCIe 3.0, so each pool should be able to host either:

• 2 x 16 lane PCIe 2.0 card
• 4 x 8 lane PCIe 2.0 cards
• 8 x 4 lane PCIe 2.0 cards (well you won't have enough PCI slots for that, as you need at least one for the Graphics card)

Of course you can combine PCIe 3.0 and 2.0 cards. Just do the maths considering the information above.

Remember that Thunderbolt connections also use PCIe bandwidth (4 lanes of PCIe 3.0 for Thunderbolt 3). See below.

Are there any recommended slots?

The first things to verify are:

• Which generation of the PCI protocol is your card using? Is it a PCIe gen1, PCIe gen2, or PCIe gen3?
• How many lanes does it use? Is it 16, 8 or 4 lanes (or less)
• If it's a 4 lane card, is it compatible with 8 or 16 lanes slots?
• If it's a 8 lane card, is it compatible with 16 lanes slots?

And here are a few recommendations:

• Graphics cards should always be connected in slot 1 or 3.
• Other 16-lanes cards (such as the Afterburner), can be connected to the remaining 16-lanes slots (1, 3, 4 or 5). But if you have only one graphics card in slot 1, we recommend using slot 3. Indeed this slot won't count in the pool allocation, and thus will leave you more bandwidth for the remaining slots.
• 8 lanes and 4 lanes cards, prefer to use slots 6 or 7. Most AJA, Blackmagic-Design and M|Family cards should work fine in these slots. But most of these cards are also compatible with 16 lanes slots, so it should be fine to use it in the 16 lanes slots. Check the tech specs of the cards to verify.

Of course, do not use for example a 16 lane card in a 8 lane slot, as you will only have half the bandwidth. So don't put your afterburner card in slots 6, 7 or 8. ;-)

About Thunderbolt devices

Thunderbolt devices will also use the PCIe Pool Allocations. So whenever you plug or unplug a Thunderbolt device, it is a good idea to go and check the Expansion Slot Utility. The Thunderbolt devices will be shown in grey:

Each Thunderbolt BUS to which at least one device is connected will allocate 4 lanes of PCIe 3.0 (even if it's a Thunderbolt 2 device that won't use as much data). Note that:

• if you plug two Thunderbolt devices in the same BUS, only 4 lanes of PCIe 3.0 will be allocated for both devices.
• Thunderbolt devices will only be allocated to Pool B. So if you the manual configuration, make sure to leave some free bandwidth on Pool B.

Check the following article for more information on the Thunderbolt connections on the Mac Pro.

What happens if I go over the limit?

If you install too many cards, or if you manually configure the "Expansion Slot Utility" and select too many cards for one pool, you may end up with something like this:

In this case, too many PCI cards have been allocated to Pool A, and thus they may run slower, which is not recommended, even though it may be OK for various reasons. 

Also, in some cases, it could be OK if the card or TB device does not have the full number of lanes it says it needs. A few examples:

• if you connect in Thunderbolt a storage that won't use the full 4 lanes (see below about what 1 lane represents in terms of data transfer)
• if you are using a video card that can do 8K, but you only use SD video... You will only use a fraction of the bandwidth.
• if you are using a two input video card, but only use one of the two, in theory your card could be 2 times slower than what it allows.
• if you connect a Thunderbolt 2 device, 4 lanes of PCIe 3.0 will be allocated, even though it can only use less.

But that's theory, and again we do not recommend this. If you do, make sure that you test this extensively, and try to test a configuration where all devices will use the maximum bandwidth that they can.

How much data is 1 lane, what does that mean?

The Mac Pro processors use the third generation of PCIe (PCIe 3.0, or PCIe gen3). With that version of PCIe, each lane can deliver 985 MB/s. So just to keep things easy (even if not entirely precise), you can remember that each lane represents 1Gb/s.

This is important when considering the limits above. Indeed, if you connect a storage to your computer  in Thunderbolt. It will "allocate" 4 lanes of PCIe 3.0, so capable of doing 4Gb/s. The question is: does your storage do that? If yes, then "lucky" you, and it will use the whole 4 lanes of PCIe 3.0. If not, then it means it will leave some extra bandwidth for the other devices.

Examples of PCIe cards

The Afterburner card is a PCIe 3.0 16 lane card. So this means that it will use a whole pool by itself, and thus if it's connected in ports 2, 4, 5, 6, 7 or 8, you will only have room for 2 x 8-lane PCIe 3.0  cards. Thus, if you only have one graphics card connected, you may want to connect the card in either slot 1 or slot 3, which don't count in the pool allocation.

The recent video cards with 12G connectors such as the AJA Kona 5, Blackmagic-Design Decklink 8K Pro or our new M|Family cards, are 8 lane PCIe 3.0 cards. Previous models are usually PCIe 2.0 cards with either 8 or 4 lane, depending on the model and the number of inputs and outputs. The article of our help center about the supported devices lists the type of PCI connection each card uses.

Some cards manufacturers (such as AJA) have more specific information about their cards, so it may be worth checking their website.