Interrupts from external devices in the x86 system. Part 3. Setting up interrupt routing in the chipset using the example of coreboot

  • Tutorial
We continue to consider setting interrupts from external devices on the x86 system.
In Part 1 ( Evolution of Interrupt Controllers ), we looked at the theoretical foundations of interrupt controllers and general terms, and in Part 2 ( Linux kernel boot options ), we looked at how the OS chooses between controllers in practice. In this part we will look at how the BIOS configures IRQ routing to the interrupt controllers in the chipset.

No modern BIOS development companies (AwardBIOS / AMIBIOS / Insyde) disclose the source code of their programs. But fortunately there is a Coreboot - a project to replace proprietary BIOS with free software. In its code, we will see how the routing of interrupts in the chipset is configured.




Theory


To start refresh and supplement their theoretical knowledge. In Part 1, we have identified a common interrupt path for the PIC and APIC case.

PIC:



APIC:



In these figures, the mapping PCI device → PIR is depicted in the abstract, in fact it is somewhat more complicated. In reality, each PCI device has 4 interrupt lines (INTA #, INTB #, INTC #, INTD #). Each PCI device (device) can have up to 8 functions (functions), and each function already has one INTx # interrupt. Which line of INTx # each function of the device will pull is either fixed in the hardware or determined by the configuration of the device.



In essence, functions are separate logical blocks. For example, in one PCI device there may be the Smbus controller function, the SATA controller function, the LPC bridge function. On the OS side, each function is as a separate device with its own PCI Config configuration space.

In the simplest (and most common) case in a PCI device, there is only one function, the interruption from which goes along the INTA # line. But in general, the device can even have more than 4 functions (as we said before 8), then some of them will have to be planted on one INTx line # (PCI interrupts can divide the line). Also, for PCI devices included in the chipset, writing to special registers often makes it possible to specify which functions use which INTx # lines (and whether they use it at all).

By systematizing our knowledge, we denote the path (routing) of interruptions from any PCI function through INTx # → PIRQy → IRQz, where:

  • INTx # - INT # line (INTA #, INTB #, INTC #, INTD #) PCI device that the function will use
  • PIRQy - the PIRQ line (PIRQA, PIRQB, ...) from the PIR to which the INTx line is connected #
  • IRQz - IRQ line (0, 1, 2, ...) on the interrupt controller (APIC / PIC), which is connected to the PIRQy line

Why can't we just connect INTA # → PIRQA, INTB # → PIRQB, ... everywhere?


Why bother with setting up routing at all? Suppose we decided not to bother and get all the interrupt lines from all PCI devices to the same PIRQ lines. So to speak:

  • INTA # → PIRQA
  • INTB # → PIRQB
  • INTC # → PIRQC
  • INTD # → PIRQD

As we have said above, the most common case is when a PCI device has one function, and its interruption is initiated on the INTA # line (because why should the device developer start it differently?). So if we suddenly decide to start all the lines as we wrote, then almost all interrupts from the devices will be divided into PIRQA lines. Suppose she wound up on IRQ16. Then, each time the processor is informed that an interrupt has occurred on the IRQ16 line, it will have to interrogate the drivers of all devices connected to the IRQ16 (PIRQA) line to see if they have an interrupt for it. If there are many such devices, this naturally does not speed up the response of the system to an interrupt. And the lines PIRQB-PIRQD in this case will be mostly idle. For clarity, the picture illustrates the problem:



But everything could have been done like this:



The picture was a bit confusing, but the point is that we simply connect the INTx lines here with PIRQy by round-robin (PIRQA, PIRQB, PIRQC, PIRQD, PIRQA, PIRQB, PIRQC, PIRQD, PIRQA, PIRQB, PIRQC, PIRQD,. ..)

It should be noted that here it is necessary to take into account not only that the same number of PCI functions be loaded on each PIRQ line. After all, some functions can create interrupts very rarely, and some constantly (for example, an Ethernet controller). In this case, even the allocation of a separate PIRQ line for interruptions from such a function can be quite justified.

Based on the foregoing, the BIOS developer, among other things, has the task of making the PIRQ lines evenly loaded with interrupts.

What should the BIOS do?


Systematized in the picture:



  • 1) Specify which line of INTx # is pulled by each of the functions of PCI devices
    For external PCI devices, this item is not performed, but it may well be for PCI functions of devices included in the chipset.
  • 2) Configure the mapping of INTx # → PIRQy for each of the PCI devices.
    It is worth noting that the PIRQy signals may be larger than the standard four (PIRQA, PIRQB, PIRQC, PIRQD). For example 8: PIRQA-PIRQH.

PIRQy signals go on the IRQz line of the selected interrupt controller (APIC / PIC). Since we want to support all possible methods of loading (see part 2 ), we need to fill in both mappings:

  • 3a) Fill in the PIRQy → IRQz1 mapping for PIR → I / O APIC communications
    But usually this is not necessary, since the PIRQy lines are fixed on the APIC line. A common solution is PIRQA → IRQ16, PIRQB → IRQ17, ... The simplest solution is because Putting the PIRQy lines on the controller lines ≥16 you can not worry about conflicts with inseparable interrupts from ISA devices.
  • 3b) Fill in the PIRQy → IRQz2 mapping for PIR → PIC communication
    This must be provided in case we use routing through the PIC controller. There is no such unambiguous solution as in the case of APIC, because in the case of PIC, one should remember about the possibility of conflicts with unshareable interruptions from ISA devices.

The last fourth point is necessary to help the OS in determining the routing interrupts. These registers are usually not used by the device itself.

  • 4) Fill in the Interrupt Line / Interrupt Pin registers for each PCI function.
    In general, the Interrupt Pin register is automatically filled and is usually Read-Only, so you will most likely need to fill only the Interrupt Line register. This should be provided in case we use routing through the PIC controller without providing the OS with any table about interrupt routing (see again part 2 ). If the tables are provided and this mapping is consistent with the routing tables ($ PIR / ACPI), then the OS often leaves it.

It should be noted that we do not touch the $ PIR / MPtable / ACPI tables yet and consider how to configure the registers of the chipset in terms of routing interrupts before transferring control to the system loader. Interrupt tables are a topic for a separate article (possibly a future one).

So, the theoretical foundations have been studied, let's finally start practicing!

Practice


As an example, for articles in this series I use a customized motherboard with an Intel Haswell i7 processor and a LynxPoint-LP chipset. On this board, I launched the coreboot in conjunction with SeaBIOS. Coreboot provides hardware-dependent initialization, and the load (payload) SeaBIOS for it provides a BIOS interface for operating systems. In this article, I will not describe the process of configuring coreboot, but will only try to show with an example what kind of BIOS settings should I make in the chipset for routing IRQ interrupts from external devices.

Since the coreboot project is being actively developed so that the article is always relevant, we will consider the code using the example of the latest fixed version 4.9 (release 2018-12-20).

The closest motherboard to mine is the Google Beltino with the Panther variation. The main folder for this motherboard is the folder “src \ mainboard \ google \ beltino” . All settings and code specific to this board are concentrated here.

So, let's begin to understand where the above items are configured:

1) Specify which line of INTx # pulls each of the functions of PCI devices


This information is defined in the file “src / mainboard / google / beltino / romstage.c” in the rcba_config structure via the DxxIP (Device xx Interrupt Pin Register (IP)) registers. This register shows which pin INTx # (A / B / C / D) each of the functions of the device displays an interrupt.

Possible options (see the file “src / southbridge / intel / lynxpoint / pch.h” ):

0h = No interrupt
1h = INTA#
2h = INTB#
3h = INTC#
4h = INTD#

It is assumed that several functions use the same pin.

It is assumed that functions may not use a pin for interrupts (No interrupt).
Everything as we saw in the figure at the beginning of the article.

The full code is responsible for the item indicated by us:

/* Device interrupt pin register (board specific) */
RCBA_SET_REG_32(D31IP, (INTC << D31IP_TTIP) | (NOINT << D31IP_SIP2) |
		(INTB << D31IP_SMIP) | (INTA << D31IP_SIP)),
RCBA_SET_REG_32(D29IP, (INTA << D29IP_E1P)),
RCBA_SET_REG_32(D28IP, (INTA << D28IP_P1IP) | (INTC << D28IP_P3IP) |
		(INTB << D28IP_P4IP)),
RCBA_SET_REG_32(D27IP, (INTA << D27IP_ZIP)),
RCBA_SET_REG_32(D26IP, (INTA << D26IP_E2P)),
RCBA_SET_REG_32(D22IP, (NOINT << D22IP_MEI1IP)),
RCBA_SET_REG_32(D20IP, (INTA << D20IP_XHCI)),

For a better understanding, consider a few examples:

Example 1:

In the 0x1d device (29 in the decimal system) one function (EHCI controller).

In this case, assign an interrupt to INTA #.

00: 1d.0 - INTA #

RCBA_SET_REG_32(D29IP, (INTA << D29IP_E1P)),

Example 2:
The device 0x1f (31 in the decimal system) has the functions Thermal Sensor controller (00: 1f.6), SATA controller 2 (00: 1f.2), SMBus controller (00: 1f.3), SATA controller 1 ( 00: 1f.2). We want to use only SMBus controller, SATA controller 1 and Thermal Sensor controller.

00: 1f.2 - INTA # (SATA controller 1)
00: 1f.3 - INTB # (SMBus controller)
00: 1f.2 - No interrupt (SATA controller 2 is not used)
00: 1f.6 - INTC # (Thermal Sensor controller)

For this configuration, write:

RCBA_SET_REG_32(D31IP, (INTC << D31IP_TTIP) | (NOINT << D31IP_SIP2) | (INTB << D31IP_SMIP) | (INTA << D31IP_SIP)),

Example 3:

In one Device, the number of functions we need is greater than 4. In the 0x1c device, each function is responsible for the PCI Express port. In order for the ports 0-5 to work, and the interrupts are evenly distributed between the lines, you can make this setting:

00: 1c.0 - INTA # (PCI Express Port 0)
00.1c.1 - INTB # (PCI Express Port 1)
00.1c. 2 - INTC # (PCI Express Port 2)
00.1c.3 - INTD # (PCI Express Port 3)
00.1c.4 - INTA # (PCI Express Port 4)
00.1c.5 - INTB # (PCI Express Port 5)
00.1 c.6 - No interrupt (port not used)
00.1c.7 - No interrupt (port not used)

RCBA_SET_REG_32(D28IP, (INTA << D28IP_P1IP) | (INTB << D28IP_P2IP) | (INTC << D28IP_P3IP) | (INTD << D28IP_P4IP) | (INTA << D28IP_P5IP) | (INTB << D28IP_P6IP) | (NOINT << D28IP_P7IP) | (NOINT << D28IP_P8IP)),

2) Configure the mapping INTx # → PIRQy for each of the PCI devices


This information is also defined in the “src \ mainboard \ google \ beltino \ romstage.c” file
in the rcba_config structure, but through the DxxIR (Device xx Interrupt Route Register) registers.

The information in this register shows to which PIRQx line (A / B / C / D / E / F / G / H) each of the INTx # interrupt lines is connected.

/* Device interrupt route registers */
RCBA_SET_REG_32(D31IR, DIR_ROUTE(PIRQG, PIRQC, PIRQB, PIRQA)),/* LPC */
RCBA_SET_REG_32(D29IR, DIR_ROUTE(PIRQD, PIRQD, PIRQD, PIRQD)),/* EHCI */
RCBA_SET_REG_32(D28IR, DIR_ROUTE(PIRQA, PIRQB, PIRQC, PIRQD)),/* PCIE */
RCBA_SET_REG_32(D27IR, DIR_ROUTE(PIRQG, PIRQG, PIRQG, PIRQG)),/* HDA */
RCBA_SET_REG_32(D22IR, DIR_ROUTE(PIRQA, PIRQA, PIRQA, PIRQA)),/* ME */
RCBA_SET_REG_32(D21IR, DIR_ROUTE(PIRQE, PIRQF, PIRQF, PIRQF)),/* SIO */
RCBA_SET_REG_32(D20IR, DIR_ROUTE(PIRQC, PIRQC, PIRQC, PIRQC)),/* XHCI */
RCBA_SET_REG_32(D23IR, DIR_ROUTE(PIRQH, PIRQH, PIRQH, PIRQH)),/* SDIO */

Example 1:

Device 0x1c (28 in decimal) are PCIe ports, as we have already figured out.

We produce a “direct” connection:

  • INTA # → PIRQA
  • INTB # → PIRQB
  • INTC # → PIRQC
  • INTD # → PIRQD

RCBA_SET_REG_32(D28IR, DIR_ROUTE(PIRQA, PIRQB, PIRQC, PIRQD))

Example 2:

Device 0x1d (29 in decimal) - one function (EHCI controller) on INTA #, the remaining lines are not used.

Connect the INTA # line with the PIRQD:

RCBA_SET_REG_32(D29IR, DIR_ROUTE(PIRQD, PIRQD, PIRQD, PIRQD))

In this case, only the first PIRQD entry (for INTA #) makes sense, the others do not make sense.

3a) Fill in the mapping PIRQy → IRQz1 (PIR → APIC)


As we have said, the mapping is often fixed here, and this case is no exception.

  • PIRQA → IRQ16
  • PIRQB → IRQ17
  • ...
  • PIRQH → IRQ23

3b) Fill in the mapping PIRQy → IRQz2 (PIR → PIC)


In coreboot, the content for filling these registers is defined in the devicetree.cb file in the motherboard folder "src \ mainboard \ google \ beltino \".

devicetree.cb (the name devicetree is for communication with a similar concept in the Linux kernel, and “cb” is an abbreviation of coreboot) is a special file that reflects the configuration of this motherboard: which processor, chipset are used, which devices are on them, which off, etc. In addition, this file may also specify specific information for the configuration of the chipset. This is exactly the case we need:

register "pirqa_routing" = "0x8b"
register "pirqb_routing" = "0x8a"
register "pirqc_routing" = "0x8b"
register "pirqd_routing" = "0x8b"
register "pirqe_routing" = "0x80"
register "pirqf_routing" = "0x80"
register "pirqg_routing" = "0x80"
register "pirqh_routing" = "0x80"

These lines set the mapping PIRQy → IRQz2. In the code after parsing the devicetree.cb file, they are transformed into the variables “config-> pirqX_routing”.

The variable “config-> pirqa_routing = 0x8b” will mean that the PIRQA is connected to the IRQ11 interrupt line (0x0b = 11) of the PIC controller, but the most severe bit (which is 0x80) means that the interrupt routing is not performed. Honestly speaking in my experience, this is a mistake, by default it is necessary to enable routing to the PIC, the operating system itself will be able to switch to I / O APIC by setting this bit to 1 if needed.

That is, in this case it would be better to write:

register "pirqa_routing" = "0x0b"
register "pirqb_routing" = "0x0a"
register "pirqc_routing" = "0x0b"
register "pirqd_routing" = "0x0b"
register "pirqe_routing" = "0x80"# not used
register "pirqf_routing" = "0x80"# not used
register "pirqg_routing" = "0x80"# not used
register "pirqh_routing" = "0x80"# not used

We did not enable the last 4 interrupts, since IRQ0 interrupt is always used under the system timer and is clearly inaccessible (see General IBM-PC Compatible Interrupt Information ).

But if we take a closer look at point 2), we will see that some PCI devices use PIRQE-PIRQH lines, so leave them unconnected the right path to non-working devices.

So it is better to write something like this:
register "pirqa_routing" = "0x03"
register "pirqb_routing" = "0x04"
register "pirqc_routing" = "0x05"
register "pirqd_routing" = "0x06"
register "pirqe_routing" = "0x0a"
register "pirqf_routing" = "0x0b"
register "pirqg_routing" = "0x0e"
register "pirqh_routing" = "0x0f"


The actual filling of the corresponding registers takes place in the src \ southbridge \ intel \ lynxpoint \ lpc.c file in the pch_pirq_init function.

Fragment of code responsible for filling registers:

/* Get the chip configuration */config_t *config = dev->chip_info;
pci_write_config8(dev, PIRQA_ROUT, config->pirqa_routing);
pci_write_config8(dev, PIRQB_ROUT, config->pirqb_routing);
pci_write_config8(dev, PIRQC_ROUT, config->pirqc_routing);
pci_write_config8(dev, PIRQD_ROUT, config->pirqd_routing);
pci_write_config8(dev, PIRQE_ROUT, config->pirqe_routing);
pci_write_config8(dev, PIRQF_ROUT, config->pirqf_routing);
pci_write_config8(dev, PIRQG_ROUT, config->pirqg_routing);
pci_write_config8(dev, PIRQH_ROUT, config->pirqh_routing);

Register address constants are described in the same pch.h file .

#define PIRQA_ROUT		0x60#define PIRQB_ROUT		0x61#define PIRQC_ROUT		0x62#define PIRQD_ROUT		0x63#define PIRQE_ROUT		0x68#define PIRQF_ROUT		0x69#define PIRQG_ROUT		0x6A#define PIRQH_ROUT		0x6B

Mapping PIRQy → IRQz2 for this chipset is written to the PCI LPC device (address 00: 1f.0) in the PIRQy_ROUT registers. It should be noted that often not all 15 IRQz2 lines on the PIC are allowed for use, but only a part (for example, 3,4,5,6,7,9,10,11,12,14,15). In the description of these registers there should be information about which IRQs are available for assigning interrupts from PIRQ lines to them. So the mapping proposed by us is possible only if PIRQ assignment on the IRQ3, IRQ4, IRQ5, IRQ6, IRQ10, IRQ11, IRQ14, IRQ15 lines is available. But if we carefully look at the comment before the pch_pirq_init function, we will see that it is:

/* PIRQ[n]_ROUT[3:0] - PIRQ Routing Control
 * 0x00 - 0000 = Reserved
 * 0x01 - 0001 = Reserved
 * 0x02 - 0010 = Reserved
 * 0x03 - 0011 = IRQ3
 * 0x04 - 0100 = IRQ4
 * 0x05 - 0101 = IRQ5
 * 0x06 - 0110 = IRQ6
 * 0x07 - 0111 = IRQ7
 * 0x08 - 1000 = Reserved
 * 0x09 - 1001 = IRQ9
 * 0x0A - 1010 = IRQ10
 * 0x0B - 1011 = IRQ11
 * 0x0C - 1100 = IRQ12
 * 0x0D - 1101 = Reserved
 * 0x0E - 1110 = IRQ14
 * 0x0F - 1111 = IRQ15
 * PIRQ[n]_ROUT[7] - PIRQ Routing Control
 * 0x80 - The PIRQ is not routed.
 */

4) Fill in the Interrupt Line / Interrupt Pin registers for each PCI function


In the PCI configuration space (each PCI function has a standard) there are 2 registers of interest to us:

  • 3Ch: Interrupt Line — cюда необходимо записать номер IRQz2 (число от 0 до 15), номер прерывания, которое в итоге дёргает функция при использовании PIC контроллера
  • 3Dh: Interrupt Pin — показывает какую линию INTx# (A/B/C/D) использует функция

Let's start with the last one. The Interrupt Pin register will be filled automatically based on the chipset settings (DxxIP registers) made by us in step 1 and will be Read-Only.

So it remains only to fill the Interrupt Line register with an IRQz2 interrupt for each PCI function.

Knowing the mapping PIRQy → IRQz2 (item 3b), and mapping INTx # → PIRQy (item 2), you can easily fill the Interrupt Line register for each function, knowing what kind of INTx # interrupt it uses (item 1).

In coreboot, the Interrupt Line registers are also filled in the src \ southbridge \ intel \ lynxpoint \ lpc.c file in the pch_pirq_init function:

/* Eric Biederman once said we should let the OS do this.
 * I am not so sure anymore he was right.
 */for (irq_dev = all_devices; irq_dev; irq_dev = irq_dev->next) {
	u8 int_pin=0, int_line=0;
	if (!irq_dev->enabled || irq_dev->path.type != DEVICE_PATH_PCI)
		continue;
	int_pin = pci_read_config8(irq_dev, PCI_INTERRUPT_PIN);
	switch (int_pin) {
	case1: /* INTA# */ int_line = config->pirqa_routing; break;
	case2: /* INTB# */ int_line = config->pirqb_routing; break;
	case3: /* INTC# */ int_line = config->pirqc_routing; break;
	case4: /* INTD# */ int_line = config->pirqd_routing; break;
	}
	if (!int_line)
		continue;
	pci_write_config8(irq_dev, PCI_INTERRUPT_LINE, int_line);
}

For some reason, this code assumes that the mapping is anyway INTA # → PIRQA, INTB # → PIRQB, INTC # → PIRQC, INTD # → PIRQD. Although, in fact, we have seen that it may be different (see point 2).

In general, "Eric Biederman once said," and we scattered everything:

$ grep "Eric Biederman once said" -r src/
src/southbridge/intel/fsp_bd82x6x/lpc.c:        /* Eric Biederman once said we should let the OS do this.
src/southbridge/intel/i82801gx/lpc.c:   /* Eric Biederman once said we should let the OS do this.
src/southbridge/intel/i82801ix/lpc.c:   /* Eric Biederman once said we should let the OS do this.
src/southbridge/intel/lynxpoint/lpc.c:  /* Eric Biederman once said we should let the OS do this.
src/southbridge/intel/sch/lpc.c:        /* Eric Biederman once said we should let the OS do this.

In general, the coreboot does not care much about supporting legacy interrupt modes. So you shouldn't be surprised at such an error. When booting a modern OS, this will not hurt you, but if you suddenly need to boot Linux with the “acpi = off nolapic” options, then this is unlikely to be done.

Conclusion


In conclusion, we will repeat the typical information that must be configured in the chipset for routing PCI interrupts:

  1. Specify which line of INTx # pulls each of the functions of PCI devices
  2. Configure INTx # → PIRQy mapping for each PCI device
  3. Fill in the PIRQy → IRQz1 mapping (PIR → APIC) and the PIRQy → IRQz2 mapping (PIR → PIC)
  4. Fill in the Interrupt Line / Interrupt Pin registers of the PCI configuration space for each PCI function.

Also popular now: