Developed by PCI-SIG®, PCI Express® (PCIe®) is a high-speed serial computer expansion bus standard that has been used in data-intensive applications such as data centers, AI/machine learning, HPC, automotives, IoT, and military/aerospace. Since the release of PCIe 5.0 and 6.0 in 2019 and 2022 respectively, PCIe has become even more common among modern motherboards and SSDs. However, not all PCIe designs are made equally — which is why we’ll be shedding light on key considerations and methodologies to help you optimize your design and test processes today.
PCIe 1.0 was first introduced in 2003 at 2.5 GT/s. In contrast, PCIe 7.0 that is anticipated to be released in 2025 features an impressive speed of 128 GT/s, showing the remarkable progress that PCI-SIG has made in pushing the boundaries of data transfer rates. Innovations such as the transition from 8b/10b to 128/132 encoding between PCIe 2.0 and 3.0, as well as the adoption of PAM4 (Pulse Amplitude Modulation 4) in PCIe 6.0 have allowed PCI-SIG to double data rates with each generational release; spaced out over an average of three years each time.
Figure 1: Evolution of PCIe technology
Comparing key differences across the last three PCIe iterations, we can see that the Nyquist frequency doubles between PCIe 4.0 and 5.0, while remaining unchanged in PCIe 6.0. However, the adoption of a different modulation scheme allowed PCIe 6.0 to feature a double data rate regardless. The insertion loss channel also increased between PCIe 4.0 to 5.0 while decreasing in PCIe 6.0, which was seen as a beneficial tradeoff for a doubled data rate.
The reference CTLE and DFE has also seen increase across all iterations to accommodate higher insertion loss or noise levels. Additionally, minimum scope bandwidth requirements have also increased. Calibrating receiver eye and testing in PCIe 4.0 only requires a minimum scope bandwidth of 25GHz. That threshold moves up to 33GHz for transmitter and receiver testing in PCIe 5.0, although receiver calibration and link EQ setup for the same standard requires 50GHz. For PCIe 6.0, 50GHz becomes mandatory for both calibration and testing.
Figure 2: PCIe layers
First, lets understanding the roles of PCIe’s three core hardware layers:
Above the three core layers of TL, DLL, and PHY are software and operating system layers responsible for drivers and configuration tasks. While the software and operating system layers are also responsible for the overall functionality of PCIe devices, they are considered to be separate from PCIe hardware layers and are typically discussed within the context of software architecture and device management rather than the PCIe protocol itself.
Figure 3: PCIe physical layer
In the PHY layer, a single PCIe link consists of two unique connection differential pairs between two point-to-point connections, supporting synchronization through embedded clocks. This architecture facilitates robust communication with support for SRIS and SRNS clocking schemes. Additionally, PCIe allows for various lane widths, including x1, x2, x4, x8, x12, x16, and x32, thus catering to different interconnection requirements.
Figure 4: Parameters for PCIe 4.0, 5.0, 6.0
The two key specifications in PCIe are Base and CEM. Both specifications differ significantly in test point definition, methodology, and degree of formality.
Whether you should follow the Base or CEM specification depends on your product's nature and customer base. Chip manufacturers primarily adhere to the Base specification, focusing on silicon-level testing and validation. End-product manufacturers follow the CEM specification, ensuring compliance with PCIe connector standards. There are also sub-specifications available for form factors beyond the Base and CEM specifications. which we will not cover in this article.
The PCIe Compliance Program oversees the certification of PCIe products and ensures that the stringent PCI-SIG standards are met. Products must undergo testing either through PCI-SIG workshops or at authorized test labs to be certified and will be added to the PCI-SIG Integrators List upon successfully passing all Gold Suite tests and achieving an 80% passing rate for interoperability testing.
Currently, the Compliance Program extends only to PCIe 4.0 and 5.0 products, and only CEM form factors are supported for the official Integrators List certification testing. If you are looking to test other form factors, you will need to bring the right adapters to convert them to CEM during testing.
Test required for various product types include:
Figure 5: Tests required for different product types
PCIe transmitter testing focuses on validating signal integrity and quality across various data rates.
The lanes to be tested depend on the configuration of the system. All transmitter presets for 8, 16, and 32 GT/s must be tested on lane 0 — the primary lane used for communication and configuration in a PCIe link. Signal quality and jitter tests are also conducted on lane 0, as well as on the adjacent lane (N-1) if present by using one qualifying preset. Configurations of greater than x4 will require lanes 0, N-1 and N/2-1 to be tested. The qualifying preset ensures that the transmitter is configured consistently for all lanes involved during testing, allowing for accurate comparison and evaluation of signal quality across lanes.
For example,
The RefCLK undergoes individual testing only for 32 GT/s systems. For all lower data rates, the RefCLK is indirectly tested while testing the transmitter signal quality. This is why for speeds of 16 GT/s or lower, a four-channel oscilloscope, also known as a two-volt test configuration, is necessary for comprehensive testing.
PLL peaking and bandwidth are tested on lane 0 only using Pulse Width Jitter (PWJ) and/or P7 compliance pattern.
Figure 6: Official PCI-SIG test fixtures that are required for PCIe CEM transmitter tests
The objective of receiver link equalization (Link EQ) testing is to guarantee that the receiver under test can achieve a Bit Error Rate (BER) of 1E-12 with zero bit errors. The PCIe receiver Link EQ test procedure is as follows:
Figure 7: Minimum and maximum eye height and eye width requirements for PCIe 4.0 and PCIe 5.0 during calibration in PCIe receiver link equalization test
Watch the full webinar to see the test setup case studies in action
Beyond the physical layer, PCIe testing extends to the link and transaction layers, which address error handling and protocol compliance. The primary objective of link and transaction layer tests is to verify error handling at each layer, starting from the logical physical layer upwards. These tests aim to validate the implementation of capabilities and ensure their proper functioning.
Link and transaction layer tests include:
While the electrical PHY layer measures analog characteristics of the device to ensure proper signal transmission and reception, the logical PHY layer refers to the protocol used in the physical layer to adjust the electrical signals, which is actually a protocol mechanism. Hence, the logical PHY layer tests mentioned above are conducted with protocol test tools instead of electrical test tools.
Link layer tests in PCIe focus on evaluating various mechanisms to ensure proper data transmission and recovery across the PCIe link. These tests ensure that the link operates within specified parameters, including data rates, error detection, and error correction mechanisms.
Mechanisms that are generally tested in the link layer include:
In the table below, you will find more examples of tests for the link layer. These tests are run with a Protocol Test Card (PTC) or an exerciser card.
Figure 8: Examples of PCIe link layer tests
Transaction layer tests involve checking for errors in the transmission of the Transaction Layer Packets (TLPs). These tests cover scenarios like incorrect information within the TLP, ensuring request completion. Tests to detect poisoned TLP, wrong TLP sequence numbers, and nullified TLP are mandatory for PCIe compliance testing, while bad EndPoint Cyclic Redundancy Check (ECRC) detection is optional.
Figure 9: Examples of PCIe transaction layer tests
When all connected devices in a PCIe link can support data rates of PCIe 3.0 or higher, link equalization will take place to optimize the connection and establish the most stable PCIe link at a higher data rate. As such, equalization tests must be conducted from PCIe 3.0 onward for all data speeds that the device supports.
The second step ensures that bad sets of coefficients are rejected and good sets of coefficients are reflected, and should be done for each data rate supported by the device.
Figure 10: Examples of PCIe equalization tests
Watch the full webinar for more details on how PCIe 5.0 and 6.0 equalization tests differ
Lane margining is a software mechanism that was introduced at PCIe 4.0 as an optional test that does not affect inclusion in the Integrators List. However, it has since become mandatory from PCIe 5.0 onwards.
Lane margining requires different steps to be conducted at different sampling points within the receiver to ensure proper implementation. When the host requests for margin information from the endpoint, verification is done by examining the numbers returned and how they relate to the margin.
While lane margining test is defined in the Electrical (PHY) Test Specification, it is run with the link and transaction tests for add-in cards in three different ways:
In system lane margining testing, electrical characteristics of the endpoint are changed and the resulting margins reported by the system will be measured. If a retimer or switch is present in the system, the test needs to measure the downstream facing retimer port connected to the slot or switch downstream port.
As the first PCIe Authorized Test Lab approved by PCI-SIG®, GRL will help you comply with PCIe specifications from 1.0 to 6.0. Speak with our experts to learn more. Still unsure about which test solutions you should use? Watch our full on-demand webinar to find out all you need to know about PCIe testing.