Mac Pro (2023 - M2 Ultra) - Limits and possibilities of PCIe slots

The 2023 Mac Pro allows users to directly connect PCIe cards. Apple gives some recommendations on how to install and configure them and which cards can be installed in the Mac Pro. Below are some more recommendations to help you get the most out of your Mac Pro.

How to open the "Expansion Slot Utility"?

As explained in the article, to verify that your card are properly installed, you should open the Expansion Slot Utility. To do that, go to the Apple menu  > About this Mac, click on "More Info", and then click the Info button (i) in the PCI cards section to open this window (© Apple):

Then click on "Expansion Slot Utility". This should open a window where you can see (at the bottom) if you are overusing the PCIe pools or not (© Apple):

By default, macOS will automatically configure the PCIe Pools depending on the cards that you connect, and we generally recommend leaving this to automatic. But in some cases you may want to configure this manually (for example, if you want to be sure that one card will always have its required bandwidth).

What are the specifications of the PCIe slots?

If you look at the expansion slot utility, you will see these are the specs of each slots:

• Slot 1: gen4, up to x16 lanes
• Slot 2: gen4, up to x16 lanes
• Slot 3: gen4, up to x8 lanes
• Slot 4: gen4, up to x8 lanes
• Slot 5: gen4, up to x8 lanes
• Slot 6: gen4, up to x8 lanes
• Slot 7: gen3, up to x4 lanes

As slot 7 is a PCIe gen3 slot, and that gen3 is half the speed of gen4, we can consider that it can use 2 lanes of gen4. Thus, in total, these PCIe slots could use up to 66 lanes of PCIe gen4.

What are the limits of the M2 Ultra?

In the section above we talked about the number of lanes that the PCIe slots can accommodate (66 lanes), but unfortunately (as with the Intel Mac Pros), it does not mean you will be able to use all slots at max speed. Indeed, the M2 Ultra has a total of 32 lanes of PCIe gen4, so about half of what your PCIe slots are capable of!

Why does pool B use bandwidth even with no card connected?

 

If you look at the screenshot above, you will see that pool B is already at 88%, though just one card (the one with USB and HDMI) it is connected to it. This leaves only 2 lanes available for pool B. 

As Apple said, you could remove that card if you don't need it. As it's using 4 lanes, you now have 6 lanes available for Pool B.

So what are the remaining 10 lanes used for? Again, as Apple explains in its article, they are used for "built-in components such as SATA, Ethernet, Wi-Fi, and Bluetooth".

Note that this bandwidth is allocated in the screenshot above, in case you use all these components at their max speed, but it does not mean they are actually used, so a PCIe card could in theory use some of that bandwidth. But never all, and it's difficult to know and guarantee what you can count on, so it's risky...

Why all these slots if it seems you can't use all of them at full speed?

First, there are many cards that will only use a few lanes. Cards that would use the full speed of the slot are quite rare. Even for high usage workflows such as ours, it is quite rare that all cards will run at their full speed. But of course, in professional workflows you want to make sure no issue happen IF you start using them at full speed, so you'll have to put procedures into place to ensure it does not happen.

How does it compare with the 2019 Intel Mac Pro?

If you want to compare this with the 2019 Mac Pro, let's use the same "scale" –  and remember that  gen4 is twice as fast as gen3, so either divide by 2 the number of PCIe gen3 lanes, or check the GB/s speed instead.

	2019 Intel Mac Pro	2023 M2 Ultra Mac Pro
Bandwidth available in CPU	64 gen3 lanes (64GB/s)
Bandwidth available in PCIe slots	92 gen3 lanes (92GB/s)	66 gen4 lanes (132GB/s)
Slots configurations	Slot 1: gen3, up to x16 lanes Slot 2: gen3, up to x8 lanes Slot 3: gen3, up to x16 lanes Slot 4: gen3, up to x16 lanes Slot 5: gen3, up to x16 lanes Slot 6: gen3, up to x8 lanes Slot 7: gen3, up to x8 lanes Slot 8: gen3, up to x4 lanes	Slot 1: gen4, up to x16 lanes Slot 2: gen4, up to x16 lanes Slot 3: gen4, up to x8 lanes Slot 4: gen4, up to x8 lanes Slot 5: gen4, up to x8 lanes Slot 6: gen4, up to x8 lanes Slot 7: gen3, up to x4 lanes
Bandwidth available in CPU	64 gen3 lanes (64GB/s)	32 gen4 lanes (64GB/s)
Bandwidth used by Graphics card	16 gen3 lanes per graphics card (up to 32 gen3 lanes)	0 lanes
Bandwidth used by Thunderbolt	Depend on the devices that are connected	Independent, does not use PCIe bandwidth
Bandwidth used by other components	0 lanes	Up to 14 gen4 lanes: - 4 lanes used by the USB/HDMI card - 10 lanes used by "built-in components such as SATA, Ethernet, Wi-Fi, and Bluetooth"
Remaining "real" available bandwidth for PCIe slots	48GB/s (48 gen3 lanes)	44GB/s (22 gen4 lanes) (when not using the USB/HDMI card)
Thunderbolt bandwidth	Uses the PCIe bandwidth	32GB/s (32 gen3 lanes) (8 x Thunderbolt BUS, with each 4 lanes of PCIe gen3
Total PCIe "real" bandwidth available (added PCIe slots and Thunderbolt)	48GB/s (48 gen3 lanes)	76GB/s (38 gen4 lanes)

So, the typical available bandwidth in PCIe slots only is roughly similar between the 2 models but if you add the available bandwidth in Thunderbolt, the 2023 M2 Ultra Mac Pro has much more bandwidth to get signals in and out.

It's also important to remember that the M2 Ultra Mac Pro won't use PCIe bandwidth for graphics cards as the GPU is in the chip (we could discuss about the questions around this in another article)

Of course, it would be great to have more PCIe bandwidth available directly in the PCIe slots, but still, it's already a lot of bandwidth available, and the fact that it is possible to receive 24 channels of 4K60p using only those PCIe slots (and no Thunderbolt) shows how great this is.

And finally, let's not forget that at some point of time rumors talked about an M2 Extreme chip (which apparently was 2 x M2 Ultra), which would then have had 64 lanes of PCIe gen4! It's not clear why Apple do not have that chip available (apparently it was for some cost and/or reliability issues), but we can bet that with such a hardware available, it will come some day...

Recommended configurations for multichannel 4K60p

Let's do our math again, and consider a specific use case where we want to be able to record 24 channels of 4K60p. Depending on the pixel format, different configurations are possible.

	YUV 8bit	YUV 10bit	RGBA
1 x 4K60p channel (with 64 audio channels)	1 GB/s	1,34 GB/s	2 GB/s
Bitrate for 1 card (4 x 4K60p channels)*	4 GB/s	5,36 GB/s	8 GB/s
PCIe gen4 nb of lanes for 1 card (4 x 4K60p channels)*	2 lanes	3 lanes	4 lanes
Recommended configuration for 6 cards (24 x 4K60p)	All cards to Pool A	5 cards to Pool A  1 card to Pool B	Not recommended for 24 channels. We would recommend max 16 channels, with 4 cards to pool A. It could be possible to add one more card on pool B, but to be tested.

* Current video cards from AJA, Blackmagic-Design and Deltacast can have a maximum of 4 channels of 4K60p on one card

You can use the information above as a starting point for other needs. If you are using HD60p, you can divide these numbers by 4, and if you are using 8K60p, multiply these by 4. Check your frame rate as well, and if you are using 4K30p instead, then simply divide these numbers and requirements by 2...

Are there any recommended slots?

There is only one limitation to consider, that slot 7 is only a 4 lane PCIe gen3 slot, and thus if you place a card in that slot, make sure that it does not require more lanes.

Also, Apple says that the Mac Pro comes with a card with USB and HDMI. It can be removed (you would then use your Mac with a Thunderbolt monitor, or "headless"), but it can't be placed in another slot.

The other recommendation is to first start using pool A. Indeed, as pool B will also be used by other components, it can bring some uncertainties about the actual available bandwidth, while with pool A you know you have 16 lanes of PCIe gen4.

About Thunderbolt devices

Unlike with the 2019 Intel Mac Pro, Thunderbolt devices will NOT use the PCIe pool allocation, so you won't have to worry about them using your PCIe bandwidth, and they can thus be used for storage, for example.

And actually if you are pushing your computer to the limits, and your video cards already use most of the PCIe bandwidth, it is be a good idea to consider using some of the Thunderbolt ports for storage or other usages (for example some network communication could be offloaded to Thunderbolt by using an adapter, freeing up some bandwidth for pool B).

Note that we did not try to max out Thunderbolt usage, together with a maxed out PCIe usage, so as usual, we recommend running tests to confirm how things behave in your specific workflow.

What happens if I go over the limit?

If you install too many cards, "Expansion Slot Utility" may end up showing a pool allocation going over 100%, so is it bad news?

It is definitely something to pay attention to, if you stay with an allocation under 100%, it's easy, the cards you use should just work fine, and it's of course the safest. As soon as you go over the limit, then you should definitely run some real life test before putting this in production. Use the same video formats, frame rates, codecs and destinations making sure that all devices will use the maximum bandwidth that they can.

You have to think about your usage:

• for pool A: try to measure the actual speed at which the cards will run.
• for pool B: on top of the speed of the cards, consider the potential USB, HDMI, Ethernet, SATA, Bluetooth usages.

Symptoms when you actually go over the available bandwidth for the pool

If you use too much bandwidth, here are some of the potential symptoms:

• as we may not be able to retrieve the frames quickly enough from the video card, it will start dropping or duplicating frames. You will see logs such as :

  • Input: Error: Dropped 2 video frames

  • Output: Error: Video card buffer underrun (2 frames duplicated)
• if you are using a storage that is using PCIe bandwidth (for example some storage connected in PCIe) you may also start seeing issues with the storage and thus:
  • When recording, the write performances will start falling, the "write buffer" may start filling up, and if it's really too slow, and the buffer goes over 30 seconds, recording will stop.
  • When playing out, the read performances may decrease, so frames may not be decompressed in time, and we'll start duplicating frames on the output.

PCIe gen3 vs PCIe gen4

PCIe versions in the Mac Pros

The M2 Ultra Mac Pro use the fourth generation of PCIe (PCIe 4.0, or PCIe gen4). With that version of PCIe, each lane can deliver about 2GB/s.

The Intel 2019 Mac Pro was using the previous version (PCIe 3.0, or PCIe gen3), it was half of that, so about 1GB/s.

Using a PCIe gen3 card in a PCIe gen4 slot

PCIe gen4 cards are still quite rare, but there should be no problem to use a PCIe gen3 card (or even a PCIe gen2 card) in a PCIe gen4 slot. The card will operate at the speed of PCIe gen3 (or gen2).

PCIe slots are generally backwards and forward compatible, so it means that you could in theory use a PCIe gen4 card in a PCIe gen3 slot, but it will then operate at half its required speed, which mean that some things may not work.

As usual it is recommended to test thoroughly to make sure that the bandwidth meet the requirements of your workflow. 

Disclaimer

The information above is mainly about how to use the PCIe bandwidth to get the video signal in. Remember to also check what the computer will actually be able to encode, and results may vary depending on the codec and format used. Make sure to also check RAM and storage requirements...

Softron does not warrant that these recommended configurations will always meet specific user requirements. They are intended as guidelines and are based on extensive testing that Softron has done, they can be changed without notice. Hardware, drivers, software and operating systems change regularly and we advise users and prospective users to evaluate their requirements in their specific environment/workflow using Softron demo licenses that are available for downloading from Softron's web site.