Interrupts from external devices in the x86 system. Part 2. Linux kernel boot options
In the previous section, we looked at the evolution of interrupt delivery from devices in x86 systems (PIC → APIC → MSI), the general theory, and all the necessary terms.
In this practical part, we will look at how to roll back to using outdated interrupt delivery methods in Linux, namely, consider the kernel boot options:
We will also look at the order in which the OS looks at the interrupt routing tables (ACPI / MPtable / $ PIR) and what effect it will have on adding load options:
You may have tried combinations of all these options when a device did not work due to a problem with interrupts. Let us examine what exactly they do and how they change the output / proc / interrupts.
Watch the interrupt in this article, we will be on a custom motherboard with Intel Haswell i7 with the lynxPoint-LP chipset on which the coreboot is running .
Interrupt information will be displayed via the command
Output at boot without additional options:
The / proc / interrupts file provides a table of the number of interrupts on each of the processors in the following form:
So, as it should be in a modern system, they are used for devices and drivers that support MSI / MSI-X interrupts. The remaining interrupts are routed through I / O APIC.
In a simplified way, the interrupt routing scheme can be drawn like this (the active paths are marked with red, the black ones are not used).
Support for MSI / MSI-X devices must be designated as the appropriate Capability in its PCI configuration space.
In the confirmation, we give a small fragment of the lspci output for devices for which it is indicated that they use MSI / MSI-X. In our case, these are the SATA controller (ahci interrupt), 2 ethernet controllers (interrupt eth58 * and eth59 *), the graphics controller (i915) and 2 HD Audio controllers (snd_hda_intel).
As we can see, these devices have a line either “MSI: Enable +” or “MSI-X: Enable +”.
We will begin to degrade the system. To begin, boot with the option pci = nomsi.
Thanks to this option, the MSI interrupts will become IO-APIC / XT-PIC depending on the interrupt controller used.
In this case, we still have the APIC priority interrupt controller, so the picture will be:
Output / proc / interrupts:
All MSI / MSI-X interrupts are expected to disappear. Instead, devices now use interrupt mode IO-APIC-fasteoi.
Note that earlier before enabling this option, eth58 and eth59 had 9 interrupts each! And now just one by one. After all, as we remember, without MSI, only one interrupt is available for one PCI function!
Some information from dmesg on initializing ethernet controllers:
- boot without option pci = nomsi:
- boot with option pci = nomsi
Due to the decrease in the number of interrupts per device, turning on this option can lead to a significant limitation of driver performance (this is not taking into account that interruptions via MSI are 3 times faster than through IO , according to the research of the Intel Reducing Interrupts). -APIC and 5 times faster than through the PIC).
This option disables I / O APIC. MSI interrupts can still go to all CPUs, but interrupts from devices can only go to CPU0, since the PIC is connected only to CPU0. But LAPIC works and other CPUs can work and handle interrupts.
As you can see, all IO-APIC- * interrupts turned into XT-PIC-XT-PIC, and these interrupts are routed only on CPU0. MSI interrupts remain unchanged and go to all CPU0-3.
Disables LAPIC. MSI interrupts cannot work without LAPIC, I / O APIC cannot work without LAPIC. Therefore, all interrupts from the devices will go to the PIC, and it only works with CPU0. And without LAPIC, the other CPUs will not even work in the system.
Output / proc / interrupts:
In fact, only one for the new version: “noapic pci = nomsi”. All interrupts from devices can only go to CPU0 through the PIC. But LAPIC works and other CPUs can work and handle interrupts.
One, because nothing can be combined with “nolapic”, because this option will make I / O APIC and MSI unavailable. So if you once registered the noapic nolapic boot options (or the most common variant is “acpi = off noapic nolapic”), then apparently you typed extra letters.
So, what will be from the options "noapic pci = nomsi":
Output / proc / interrupts:
How does the operating system get information about routing interrupts from devices? The BIOS prepares information for the OS in the form of:
It should be noted that to designate MSI interrupts, the BIOS does not need to do anything extra; all the above information is needed only for APIC / PIC interrupt lines.
The tables in the list above are indicated in order of priority. Consider this in more detail.
Suppose the BIOS has provided all this data and we are loading without any additional options:
In case we are loading with the noapic option :
If the ACPI table is missing or the interrupt routing functionality through ACPI is disabled using the acpi = noirq or pci = noacpi options (or ACPI is completely disabled with acpi = off ), the OS looks for the MPtable (_MP_) table for interrupt routing:
If the ACPI table is missing or the interrupt routing functionality via ACPI is disabled using acpi = noirq or pci = noacpi (or ACPI is completely disabled using acpi = off ) and if the MPtable (_MP_) table is missing (or the noapic or nolapic boot option is passed ):
If there is no $ PIR table, or it is not complete, then the operating system for guessing interrupts will look at the values of the 0x3C / 0x3D registers of the PCI device configuration space.
We summarize all of the above with the following picture:
It should be remembered that not every BIOS provides all 3 tables (ACPI / MPtable / $ PIR), so if you passed the option to the loader to refuse to use ACPI or ACPI and MPtable for interrupt routing, it’s not a fact that your system will boot.
Note 1 : in case we try to boot in the APIC mode with the acpi = noirq option and without the MPtable, the interrupt pattern will be the same as in the normal boot with the only noapic option. The operating system itself will go into PIC interrupt mode.
If we try to load without any ACPI tables at all (acpi = off) and not providing the MPtable, the picture will be like this:
This is due to the fact that without the ACPI MADT table ( Multiple APIC Description Table ) and the necessary information from the MPtable, the operating system does not know the APIC identifiers (APIC ID) for other processors and cannot work with them, but the LAPIC of the main processor works, since we did not prohibit it, and MSI interrupts can come to it. That is, it will be like this:
Note 2: in general, interrupt routing using ACPI in the case of APIC coincides with interrupt routing via the MPtable. And routing interrupts via ACPI in the case of using the PIC coincides with the routing interrupts through $ PIR. So the findings / proc / interrupts should not be different. However, in the process of research I noticed one oddity. When routing via MPtable in the output for some reason, there is a cascade interrupt "XT-PIC-XT-PIC cascade".
It is a bit strange that this happens, but the kernel documentation seems to say that this is normal.
In conclusion, we again denote the parsed options.
Interrupt Controller Selection Options:
Options for selecting the interrupt routing priority table:
In the next part, let's see how the coreboot configures the chipset for routing interrupts.
In this practical part, we will look at how to roll back to using outdated interrupt delivery methods in Linux, namely, consider the kernel boot options:
- pci = nomsi
- noapic
- nolapic
We will also look at the order in which the OS looks at the interrupt routing tables (ACPI / MPtable / $ PIR) and what effect it will have on adding load options:
- pci = noacpi
- acpi = noirq
- acpi = off
You may have tried combinations of all these options when a device did not work due to a problem with interrupts. Let us examine what exactly they do and how they change the output / proc / interrupts.
Download without additional options
Watch the interrupt in this article, we will be on a custom motherboard with Intel Haswell i7 with the lynxPoint-LP chipset on which the coreboot is running .
Interrupt information will be displayed via the command
cat /proc/interruptsOutput at boot without additional options:
CPU0 CPU1 CPU2 CPU3
0: 15 0 0 0 IO-APIC-edge timer
1: 0 1 0 1 IO-APIC-edge i8042
8: 0 0 0 1 IO-APIC-edge rtc0
9: 0 0 0 0 IO-APIC-fasteoi acpi
12: 0 0 0 1 IO-APIC-edge
23: 16 247 7 10 IO-APIC-fasteoi ehci_hcd:usb1
56: 0 0 0 0 PCI-MSI-edge aerdrv,PCIe PME
57: 0 0 0 0 PCI-MSI-edge aerdrv,PCIe PME
58: 0 0 0 0 PCI-MSI-edge aerdrv,PCIe PME
59: 0 0 0 0 PCI-MSI-edge aerdrv,PCIe PME
60: 0 0 0 0 PCI-MSI-edge aerdrv,PCIe PME
61: 0 0 0 0 PCI-MSI-edge aerdrv,PCIe PME
62: 3118 1984 972 3454 PCI-MSI-edge ahci
63: 1 0 0 0 PCI-MSI-edge eth59
64: 2095 57 4 832 PCI-MSI-edge eth59-rx-0
65: 6 18 1 1309 PCI-MSI-edge eth59-rx-1
66: 13 512 2 1 PCI-MSI-edge eth59-rx-2
67: 10 61 232 2 PCI-MSI-edge eth59-rx-3
68: 169 0 0 0 PCI-MSI-edge eth59-tx-0
69: 14 14 4 205 PCI-MSI-edge eth59-tx-1
70: 11 491 3 0 PCI-MSI-edge eth59-tx-2
71: 20 19 134 50 PCI-MSI-edge eth59-tx-3
72: 0 0 0 0 PCI-MSI-edge eth58
73: 2 1 0 152 PCI-MSI-edge eth58-rx-0
74: 3 150 2 0 PCI-MSI-edge eth58-rx-1
75: 2 34 117 2 PCI-MSI-edge eth58-rx-2
76: 153 0 2 0 PCI-MSI-edge eth58-rx-3
77: 4 0 2 149 PCI-MSI-edge eth58-tx-0
78: 4 149 2 0 PCI-MSI-edge eth58-tx-1
79: 4 0 117 34 PCI-MSI-edge eth58-tx-2
80: 153 0 2 0 PCI-MSI-edge eth58-tx-3
81: 66 106 2 101 PCI-MSI-edge snd_hda_intel
82: 928 5657 262 224 PCI-MSI-edge i915
83: 545 56 32 15 PCI-MSI-edge snd_hda_intel
NMI: 0 0 0 0 Non-maskable interrupts
LOC: 4193 3644 3326 3499 Local timer interrupts
SPU: 0 0 0 0 Spurious interrupts
PMI: 0 0 0 0 Performance monitoring interrupts
IWI: 290 233 590 111 IRQ work interrupts
RTR: 3 0 0 0 APIC ICR read retries
RES: 1339 2163 2404 1946 Rescheduling interrupts
CAL: 607 537 475 559 Function call interrupts
TLB: 163 202 164 251 TLB shootdowns
TRM: 48 48 48 48 Thermal event interrupts
THR: 0 0 0 0 Threshold APIC interrupts
MCE: 0 0 0 0 Machine check exceptions
MCP: 3 3 3 3 Machine check polls
ERR: 0
MIS: 0
The / proc / interrupts file provides a table of the number of interrupts on each of the processors in the following form:
- First column: interrupt number
- CPUx columns: interrupt counters for each processor
- Next column: interrupt type:
- IO-APIC-edge - interrupt on the front to the controller I / O APIC
- IO-APIC-fasteoi - level interrupt on I / O APIC controller
- PCI-MSI-edge - MSI interrupt
- XT-PIC-XT-PIC - interrupt on the PIC controller (see later)
- Last column: device associated with this interrupt
So, as it should be in a modern system, they are used for devices and drivers that support MSI / MSI-X interrupts. The remaining interrupts are routed through I / O APIC.
In a simplified way, the interrupt routing scheme can be drawn like this (the active paths are marked with red, the black ones are not used).
Support for MSI / MSI-X devices must be designated as the appropriate Capability in its PCI configuration space.
In the confirmation, we give a small fragment of the lspci output for devices for which it is indicated that they use MSI / MSI-X. In our case, these are the SATA controller (ahci interrupt), 2 ethernet controllers (interrupt eth58 * and eth59 *), the graphics controller (i915) and 2 HD Audio controllers (snd_hda_intel).
lspci -v00:02.0 VGA compatible controller: Intel Corporation Haswell-ULT Integrated Graphics Controller (rev 09) (prog-if 00 [VGA controller])
...
Capabilities: [90] MSI: Enable+ Count=1/1 Maskable- 64bit-
Capabilities: [d0] Power Management version 2
Capabilities: [a4] PCI Advanced Features
Kernel driver in use: i915
00:03.0 Audio device: Intel Corporation Haswell-ULT HD Audio Controller (rev 09
...
Capabilities: [60] MSI: Enable+ Count=1/1 Maskable- 64bit-
Capabilities: [70] Express Root Complex Integrated Endpoint, MSI 00
Kernel driver in use: snd_hda_intel
00:1b.0 Audio device: Intel Corporation 8 Series HD Audio Controller (rev 04)
...
Capabilities: [60] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [70] Express Root Complex Integrated Endpoint, MSI 00
Capabilities: [100] Virtual Channel
Kernel driver in use: snd_hda_intel
00:1f.2 SATA controller: Intel Corporation 8 Series SATA Controller 1 [AHCI mode] (rev 04) (prog-if 01 [AHCI 1.0])
...
Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
Capabilities: [70] Power Management version 3
Capabilities: [a8] SATA HBA v1.0
Kernel driver in use: ahci
05:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
...
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Capabilities: [70] MSI-X: Enable+ Count=10 Masked-
Capabilities: [a0] Express Endpoint, MSI 00
Kernel driver in use: igb
05:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
...
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Capabilities: [70] MSI-X: Enable+ Count=10 Masked-
Capabilities: [a0] Express Endpoint, MSI 00
Kernel driver in use: igbAs we can see, these devices have a line either “MSI: Enable +” or “MSI-X: Enable +”.
We will begin to degrade the system. To begin, boot with the option pci = nomsi.
pci = nomsi
Thanks to this option, the MSI interrupts will become IO-APIC / XT-PIC depending on the interrupt controller used.
In this case, we still have the APIC priority interrupt controller, so the picture will be:
Output / proc / interrupts:
CPU0 CPU1 CPU2 CPU3
0: 15 0 0 0 IO-APIC-edge timer
1: 0 1 0 1 IO-APIC-edge i8042
8: 0 0 1 0 IO-APIC-edge rtc0
9: 0 0 0 0 IO-APIC-fasteoi acpi
12: 0 0 0 1 IO-APIC-edge
16: 1314 5625 342 555 IO-APIC-fasteoi i915, snd_hda_intel, eth59
17: 5 0 1 34 IO-APIC-fasteoi eth58
21: 2882 2558 963 2088 IO-APIC-fasteoi ahci
22: 26 81 2 170 IO-APIC-fasteoi snd_hda_intel
23: 23 369 8 8 IO-APIC-fasteoi ehci_hcd:usb1
NMI: 0 0 0 0 Non-maskable interrupts
LOC: 3011 3331 2435 2617 Local timer interrupts
SPU: 0 0 0 0 Spurious interrupts
PMI: 0 0 0 0 Performance monitoring interrupts
IWI: 197 228 544 85 IRQ work interrupts
RTR: 3 0 0 0 APIC ICR read retries
RES: 1708 2349 1821 1569 Rescheduling interrupts
CAL: 520 554 509 555 Function call interrupts
TLB: 187 181 205 179 TLB shootdowns
TRM: 102 102 102 102 Thermal event interrupts
THR: 0 0 0 0 Threshold APIC interrupts
MCE: 0 0 0 0 Machine check exceptions
MCP: 2 2 2 2 Machine check polls
ERR: 0
MIS: 0All MSI / MSI-X interrupts are expected to disappear. Instead, devices now use interrupt mode IO-APIC-fasteoi.
Note that earlier before enabling this option, eth58 and eth59 had 9 interrupts each! And now just one by one. After all, as we remember, without MSI, only one interrupt is available for one PCI function!
Some information from dmesg on initializing ethernet controllers:
- boot without option pci = nomsi:
igb: Intel(R) Gigabit Ethernet Network Driver - version 5.0.5-k
igb: Copyright (c) 2007-2013 Intel Corporation.
acpi:acpi_pci_irq_enable: igb 0000:05:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
igb 0000:05:00.0: irq 63 for MSI/MSI-X
igb 0000:05:00.0: irq 64 for MSI/MSI-X
igb 0000:05:00.0: irq 65 for MSI/MSI-X
igb 0000:05:00.0: irq 66 for MSI/MSI-X
igb 0000:05:00.0: irq 67 for MSI/MSI-X
igb 0000:05:00.0: irq 68 for MSI/MSI-X
igb 0000:05:00.0: irq 69 for MSI/MSI-X
igb 0000:05:00.0: irq 70 for MSI/MSI-X
igb 0000:05:00.0: irq 71 for MSI/MSI-X
igb 0000:05:00.0: irq 63 for MSI/MSI-X
igb 0000:05:00.0: irq 64 for MSI/MSI-X
igb 0000:05:00.0: irq 65 for MSI/MSI-X
igb 0000:05:00.0: irq 66 for MSI/MSI-X
igb 0000:05:00.0: irq 67 for MSI/MSI-X
igb 0000:05:00.0: irq 68 for MSI/MSI-X
igb 0000:05:00.0: irq 69 for MSI/MSI-X
igb 0000:05:00.0: irq 70 for MSI/MSI-X
igb 0000:05:00.0: irq 71 for MSI/MSI-X
igb 0000:05:00.0: added PHC on eth0
igb 0000:05:00.0: Intel(R) Gigabit Ethernet Network Connection
igb 0000:05:00.0: eth0: (PCIe:5.0Gb/s:Width x1) 00:15:d5:03:00:2a
igb 0000:05:00.0: eth0: PBA No: 106300-000
igb 0000:05:00.0: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
acpi:acpi_pci_irq_enable: igb 0000:05:00.1: PCI INT B -> GSI 17 (level, low) -> IRQ 17
igb 0000:05:00.1: irq 72 for MSI/MSI-X
igb 0000:05:00.1: irq 73 for MSI/MSI-X
igb 0000:05:00.1: irq 74 for MSI/MSI-X
igb 0000:05:00.1: irq 75 for MSI/MSI-X
igb 0000:05:00.1: irq 76 for MSI/MSI-X
igb 0000:05:00.1: irq 77 for MSI/MSI-X
igb 0000:05:00.1: irq 78 for MSI/MSI-X
igb 0000:05:00.1: irq 79 for MSI/MSI-X
igb 0000:05:00.1: irq 80 for MSI/MSI-X
igb 0000:05:00.1: irq 72 for MSI/MSI-X
igb 0000:05:00.1: irq 73 for MSI/MSI-X
igb 0000:05:00.1: irq 74 for MSI/MSI-X
igb 0000:05:00.1: irq 75 for MSI/MSI-X
igb 0000:05:00.1: irq 76 for MSI/MSI-X
igb 0000:05:00.1: irq 77 for MSI/MSI-X
igb 0000:05:00.1: irq 78 for MSI/MSI-X
igb 0000:05:00.1: irq 79 for MSI/MSI-X
igb 0000:05:00.1: irq 80 for MSI/MSI-X
igb 0000:05:00.1: added PHC on eth1
igb 0000:05:00.1: Intel(R) Gigabit Ethernet Network Connection
igb 0000:05:00.1: eth1: (PCIe:5.0Gb/s:Width x1) 00:15:d5:03:00:2b
igb 0000:05:00.1: eth1: PBA No: 106300-000
igb 0000:05:00.1: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)- boot with option pci = nomsi
igb: Intel(R) Gigabit Ethernet Network Driver - version 5.0.5-k
igb: Copyright (c) 2007-2013 Intel Corporation.
acpi:acpi_pci_irq_enable: igb 0000:05:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
igb 0000:05:00.0: added PHC on eth0
igb 0000:05:00.0: Intel(R) Gigabit Ethernet Network Connection
igb 0000:05:00.0: eth0: (PCIe:5.0Gb/s:Width x1) 00:15:d5:03:00:2a
igb 0000:05:00.0: eth0: PBA No: 106300-000
igb 0000:05:00.0: Using legacy interrupts. 1 rx queue(s), 1 tx queue(s)
acpi:acpi_pci_irq_enable: igb 0000:05:00.1: PCI INT B -> GSI 17 (level, low) -> IRQ 17
igb 0000:05:00.1: added PHC on eth1
igb 0000:05:00.1: Intel(R) Gigabit Ethernet Network Connection
igb 0000:05:00.1: eth1: (PCIe:5.0Gb/s:Width x1) 00:15:d5:03:00:2b
igb 0000:05:00.1: eth1: PBA No: 106300-000
igb 0000:05:00.1: Using legacy interrupts. 1 rx queue(s), 1 tx queue(s)Due to the decrease in the number of interrupts per device, turning on this option can lead to a significant limitation of driver performance (this is not taking into account that interruptions via MSI are 3 times faster than through IO , according to the research of the Intel Reducing Interrupts). -APIC and 5 times faster than through the PIC).
noapic
This option disables I / O APIC. MSI interrupts can still go to all CPUs, but interrupts from devices can only go to CPU0, since the PIC is connected only to CPU0. But LAPIC works and other CPUs can work and handle interrupts.
CPU0 CPU1 CPU2 CPU3
0: 5 0 0 0 XT-PIC-XT-PIC timer
1: 2 0 0 0 XT-PIC-XT-PIC i8042
2: 0 0 0 0 XT-PIC-XT-PIC cascade
8: 1 0 0 0 XT-PIC-XT-PIC rtc0
9: 0 0 0 0 XT-PIC-XT-PIC acpi
12: 172 0 0 0 XT-PIC-XT-PIC ehci_hcd:usb1
56: 0 0 0 0 PCI-MSI-edge aerdrv, PCIe PME
57: 0 0 0 0 PCI-MSI-edge aerdrv, PCIe PME
58: 0 0 0 0 PCI-MSI-edge aerdrv, PCIe PME
59: 0 0 0 0 PCI-MSI-edge aerdrv, PCIe PME
60: 0 0 0 0 PCI-MSI-edge aerdrv, PCIe PME
61: 0 0 0 0 PCI-MSI-edge aerdrv, PCIe PME
62: 2833 2989 1021 811 PCI-MSI-edge ahci
63: 0 1 0 0 PCI-MSI-edge eth59
64: 301 52 9 3 PCI-MSI-edge eth59-rx-0
65: 12 24 3 178 PCI-MSI-edge eth59-rx-1
66: 14 85 6 2 PCI-MSI-edge eth59-rx-2
67: 17 24 307 1 PCI-MSI-edge eth59-rx-3
68: 70 18 8 10 PCI-MSI-edge eth59-tx-0
69: 7 0 0 23 PCI-MSI-edge eth59-tx-1
70: 15 227 2 2 PCI-MSI-edge eth59-tx-2
71: 18 6 27 2 PCI-MSI-edge eth59-tx-3
72: 0 0 0 0 PCI-MSI-edge eth58
73: 1 0 0 27 PCI-MSI-edge eth58-rx-0
74: 1 22 0 5 PCI-MSI-edge eth58-rx-1
75: 1 0 22 5 PCI-MSI-edge eth58-rx-2
76: 23 0 0 5 PCI-MSI-edge eth58-rx-3
77: 1 0 0 27 PCI-MSI-edge eth58-tx-0
78: 1 22 0 5 PCI-MSI-edge eth58-tx-1
79: 1 0 22 5 PCI-MSI-edge eth58-tx-2
80: 23 0 0 5 PCI-MSI-edge eth58-tx-3
81: 187 17 70 7 PCI-MSI-edge snd_hda_intel
82: 698 1647 247 129 PCI-MSI-edge i915
83: 438 135 16 59 PCI-MSI-edge snd_hda_intel
NMI: 0 0 0 0 Non-maskable interrupts
LOC: 1975 2499 2245 1474 Local timer interrupts
SPU: 0 0 0 0 Spurious interrupts
PMI: 0 0 0 0 Performance monitoring interrupts
IWI: 132 67 429 91 IRQ work interrupts
RTR: 3 0 0 0 APIC ICR read retries
RES: 1697 2178 1903 1541 Rescheduling interrupts
CAL: 561 496 534 567 Function call interrupts
TLB: 229 254 170 137 TLB shootdowns
TRM: 78 78 78 78 Thermal event interrupts
THR: 0 0 0 0 Threshold APIC interrupts
MCE: 0 0 0 0 Machine check exceptions
MCP: 2 2 2 2 Machine check polls
ERR: 0
MIS: 0As you can see, all IO-APIC- * interrupts turned into XT-PIC-XT-PIC, and these interrupts are routed only on CPU0. MSI interrupts remain unchanged and go to all CPU0-3.
nolapic
Disables LAPIC. MSI interrupts cannot work without LAPIC, I / O APIC cannot work without LAPIC. Therefore, all interrupts from the devices will go to the PIC, and it only works with CPU0. And without LAPIC, the other CPUs will not even work in the system.
Output / proc / interrupts:
CPU0
0: 6416 XT-PIC-XT-PIC timer
1: 2 XT-PIC-XT-PIC i8042
2: 0 XT-PIC-XT-PIC cascade
3: 5067 XT-PIC-XT-PIC aerdrv, aerdrv, PCIe PME, PCIe PME, i915, snd_hda_intel, eth59
4: 32 XT-PIC-XT-PIC aerdrv, aerdrv, PCIe PME, PCIe PME, eth58
5: 0 XT-PIC-XT-PIC aerdrv, PCIe PME
6: 0 XT-PIC-XT-PIC aerdrv, PCIe PME
8: 1 XT-PIC-XT-PIC rtc0
9: 0 XT-PIC-XT-PIC acpi
11: 274 XT-PIC-XT-PIC snd_hda_intel
12: 202 XT-PIC-XT-PIC ehci_hcd:usb1
15: 7903 XT-PIC-XT-PIC ahci
NMI: 0 Non-maskable interrupts
LOC: 0 Local timer interrupts
SPU: 0 Spurious interrupts
PMI: 0 Performance monitoring interrupts
IWI: 0 IRQ work interrupts
RTR: 0 APIC ICR read retries
RES: 0 Rescheduling interrupts
CAL: 0 Function call interrupts
TLB: 0 TLB shootdowns
TRM: 0 Thermal event interrupts
THR: 0 Threshold APIC interrupts
MCE: 0 Machine check exceptions
MCP: 1 Machine check polls
ERR: 0
MIS: 0
Combinations:
In fact, only one for the new version: “noapic pci = nomsi”. All interrupts from devices can only go to CPU0 through the PIC. But LAPIC works and other CPUs can work and handle interrupts.
One, because nothing can be combined with “nolapic”, because this option will make I / O APIC and MSI unavailable. So if you once registered the noapic nolapic boot options (or the most common variant is “acpi = off noapic nolapic”), then apparently you typed extra letters.
So, what will be from the options "noapic pci = nomsi":
Output / proc / interrupts:
CPU0 CPU1 CPU2 CPU3
0: 5 0 0 0 XT-PIC-XT-PIC timer
1: 2 0 0 0 XT-PIC-XT-PIC i8042
2: 0 0 0 0 XT-PIC-XT-PIC cascade
3: 5072 0 0 0 XT-PIC-XT-PIC i915, snd_hda_intel, eth59
4: 32 0 0 0 XT-PIC-XT-PIC eth58
8: 1 0 0 0 XT-PIC-XT-PIC rtc0
9: 0 0 0 0 XT-PIC-XT-PIC acpi
11: 281 0 0 0 XT-PIC-XT-PIC snd_hda_intel
12: 200 0 0 0 XT-PIC-XT-PIC ehci_hcd:usb1
15: 7930 0 0 0 XT-PIC-XT-PIC ahci
NMI: 0 0 0 0 Non-maskable interrupts
LOC: 2595 2387 2129 1697 Local timer interrupts
SPU: 0 0 0 0 Spurious interrupts
PMI: 0 0 0 0 Performance monitoring interrupts
IWI: 159 90 482 135 IRQ work interrupts
RTR: 3 0 0 0 APIC ICR read retries
RES: 1568 1666 1810 1833 Rescheduling interrupts
CAL: 431 556 549 558 Function call interrupts
TLB: 124 184 156 274 TLB shootdowns
TRM: 116 116 116 116 Thermal event interrupts
THR: 0 0 0 0 Threshold APIC interrupts
MCE: 0 0 0 0 Machine check exceptions
MCP: 2 2 2 2 Machine check polls
ERR: 0
MIS: 0Interrupt routing tables and options "acpi = noirq", "pci = noacpi", "acpi = off"
How does the operating system get information about routing interrupts from devices? The BIOS prepares information for the OS in the form of:
- ACPI tables (_PIC / _PRT methods)
- _MP_ tables (MPtable)
- $ PIR tables
- Registers 0x3C / 0x3D PCI device configuration space
It should be noted that to designate MSI interrupts, the BIOS does not need to do anything extra; all the above information is needed only for APIC / PIC interrupt lines.
The tables in the list above are indicated in order of priority. Consider this in more detail.
Suppose the BIOS has provided all this data and we are loading without any additional options:
- OS finds ACPI tables
- The OS executes the ACPI "_PIC" method, passing it the argument that it needs to be loaded in APIC mode. Here the method code usually saves the selected mode in a variable (for example, PICM = 1)
- To obtain interrupt data, the OS calls the ACPI method "_PRT". He internally checks the PICM variable and returns the routing for an APIC case.
In case we are loading with the noapic option :
- OS finds ACPI tables
- The OS executes the ACPI "_PIC" method, passing it the argument that it is necessary to boot in PIC mode. Here the method code usually saves the selected mode in a variable (for example, PICM = 0)
- To obtain interrupt data, the OS calls the ACPI method "_PRT". He internally checks the PICM variable and returns the routing for the PIC case.
If the ACPI table is missing or the interrupt routing functionality through ACPI is disabled using the acpi = noirq or pci = noacpi options (or ACPI is completely disabled with acpi = off ), the OS looks for the MPtable (_MP_) table for interrupt routing:
- OS does not find / look at ACPI tables
- OS finds MPtable (_MP_) table
If the ACPI table is missing or the interrupt routing functionality via ACPI is disabled using acpi = noirq or pci = noacpi (or ACPI is completely disabled using acpi = off ) and if the MPtable (_MP_) table is missing (or the noapic or nolapic boot option is passed ):
- OS does not find / look at the ACPI table
- The OS does not find / look at the MPtable table (_MP_)
- OS finds the $ PIR table
If there is no $ PIR table, or it is not complete, then the operating system for guessing interrupts will look at the values of the 0x3C / 0x3D registers of the PCI device configuration space.
We summarize all of the above with the following picture:
It should be remembered that not every BIOS provides all 3 tables (ACPI / MPtable / $ PIR), so if you passed the option to the loader to refuse to use ACPI or ACPI and MPtable for interrupt routing, it’s not a fact that your system will boot.
Note 1 : in case we try to boot in the APIC mode with the acpi = noirq option and without the MPtable, the interrupt pattern will be the same as in the normal boot with the only noapic option. The operating system itself will go into PIC interrupt mode.
If we try to load without any ACPI tables at all (acpi = off) and not providing the MPtable, the picture will be like this:
CPU0
0: 6 XT-PIC-XT-PIC timer
1: 2 XT-PIC-XT-PIC i8042
2: 0 XT-PIC-XT-PIC cascade
8: 0 XT-PIC-XT-PIC rtc0
12: 373 XT-PIC-XT-PIC ehci_hcd:usb1
16: 0 PCI-MSI-edge PCIe PME
17: 0 PCI-MSI-edge PCIe PME
18: 0 PCI-MSI-edge PCIe PME
19: 0 PCI-MSI-edge PCIe PME
20: 0 PCI-MSI-edge PCIe PME
21: 0 PCI-MSI-edge PCIe PME
22: 8728 PCI-MSI-edge ahci
23: 1 PCI-MSI-edge eth59
24: 1301 PCI-MSI-edge eth59-rx-0
25: 113 PCI-MSI-edge eth59-tx-0
26: 0 PCI-MSI-edge eth58
27: 45 PCI-MSI-edge eth58-rx-0
28: 45 PCI-MSI-edge eth58-tx-0
29: 1280 PCI-MSI-edge snd_hda_intel
NMI: 2 Non-maskable interrupts
LOC: 24076 Local timer interrupts
SPU: 0 Spurious interrupts
PMI: 2 Performance monitoring interrupts
IWI: 2856 IRQ work interrupts
RTR: 0 APIC ICR read retries
RES: 0 Rescheduling interrupts
CAL: 0 Function call interrupts
TLB: 0 TLB shootdowns
TRM: 34 Thermal event interrupts
THR: 0 Threshold APIC interrupts
MCE: 0 Machine check exceptions
MCP: 2 Machine check polls
ERR: 0
MIS: 0
This is due to the fact that without the ACPI MADT table ( Multiple APIC Description Table ) and the necessary information from the MPtable, the operating system does not know the APIC identifiers (APIC ID) for other processors and cannot work with them, but the LAPIC of the main processor works, since we did not prohibit it, and MSI interrupts can come to it. That is, it will be like this:
Note 2: in general, interrupt routing using ACPI in the case of APIC coincides with interrupt routing via the MPtable. And routing interrupts via ACPI in the case of using the PIC coincides with the routing interrupts through $ PIR. So the findings / proc / interrupts should not be different. However, in the process of research I noticed one oddity. When routing via MPtable in the output for some reason, there is a cascade interrupt "XT-PIC-XT-PIC cascade".
CPU0 CPU1 CPU2 CPU3
0: 15 0 0 0 IO-APIC-edge timer
1: 2 0 0 0 IO-APIC-edge i8042
2: 0 0 0 0 XT-PIC-XT-PIC cascade
8: 0 1 0 0 IO-APIC-edge rtc0
9: 0 0 0 0 IO-APIC-edge acpi
...
It is a bit strange that this happens, but the kernel documentation seems to say that this is normal.
Conclusion:
In conclusion, we again denote the parsed options.
Interrupt Controller Selection Options:
- pci = nomsi - MSI interrupts will become IO-APIC / XT-PIC depending on the interrupt controller used
- noapic - Disable I / O APIC. MSI interrupts can still go to all CPUs, other interrupts from devices can only go to the PIC, and it only works with CPU0. But LAPIC works and other CPUs can work and handle interrupts.
- noapic pci = nomsi - All interrupts from devices can only go to the PIC, and it works only with CPU0. But LAPIC works and other CPUs can work and handle interrupts.
- nolapic - Disables LAPIC. MSI interrupts cannot work without LAPIC, I / O APIC cannot work without LAPIC. All interrupts from the devices will go to the PIC, and it only works with CPU0. And without LAPIC, the rest of the CPU will not work.
Options for selecting the interrupt routing priority table:
- no options - routing through APIC using ACPI tables
- noapic - routing through PIC using ACPI tables
- acpi = noirq ( pci = noacpi / acpi = off ) - routing via APIC using the MPtable table
- acpi = noirq ( pci = noacpi / acpi = off ) noapic ( nolapic ) - routing through the PIC using the $ PIR table
In the next part, let's see how the coreboot configures the chipset for routing interrupts.