📜 ⬆️ ⬇️

Interrupts from external devices in the x86 system. Part 3. Setting up interrupt routing in the chipset using the example of coreboot

We continue to consider setting interrupts from external devices on the x86 system.
In Part 1 ( Evolution of Interrupt Controllers ), we looked at the theoretical foundations of interrupt controllers and general terms, and in Part 2 ( Linux kernel boot options ), we looked at how the OS chooses between controllers in practice. In this part we will look at how the BIOS configures IRQ routing to the interrupt controllers in the chipset.

No modern BIOS development companies (AwardBIOS / AMIBIOS / Insyde) disclose the source code of their programs. But fortunately there is a Coreboot - a project to replace proprietary BIOS with free software. In its code, we will see how the routing of interrupts in the chipset is configured.




Theory


To start refresh and supplement their theoretical knowledge. In Part 1, we have identified a common interrupt path for the PIC and APIC case.

PIC:



APIC:



In these figures, the mapping of the PCI device → PIR is depicted in the abstract, in fact it is somewhat more complicated. In reality, each PCI device has 4 interrupt lines (INTA #, INTB #, INTC #, INTD #). Each PCI device (device) can have up to 8 functions (functions), and each function already has one INTx # interrupt. Which line of INTx # each function of the device will pull is either fixed in the hardware or determined by the configuration of the device.



In essence, functions are separate logical blocks. For example, in one PCI device there may be the Smbus controller function, the SATA controller function, the LPC bridge function. On the OS side, each function is as a separate device with its own PCI Config configuration space.

In the simplest (and most common) case in a PCI device, there is only one function, the interruption from which goes along the INTA # line. But in general, the device can even have more than 4 functions (as we said before 8), then some of them will have to be planted on one INTx line # (PCI interrupts can divide the line). Also, for PCI devices included in the chipset, writing to special registers often makes it possible to specify which functions use which INTx # lines (and whether they use it at all).

By systematizing our knowledge, we denote the path (routing) of interruptions from any PCI function through INTx # → PIRQy → IRQz, where:


Why can't we just connect INTA # → PIRQA, INTB # → PIRQB, ... everywhere?


Why bother with setting up routing at all? Suppose we decided not to bother and get all the interrupt lines from all PCI devices to the same PIRQ lines. So to speak:


As we have said above, the most common case is when a PCI device has one function, and its interruption is initiated on the INTA # line (because why should the device developer start it differently?). So if we suddenly decide to start all the lines as we wrote, then almost all interrupts from the devices will be divided into PIRQA lines. Suppose she wound up on IRQ16. Then, each time the processor is informed that an interrupt has occurred on the IRQ16 line, it will have to interrogate the drivers of all devices connected to the IRQ16 (PIRQA) line to see if they have an interrupt for it. If there are many such devices, this naturally does not speed up the response of the system to an interrupt. And the lines PIRQB-PIRQD in this case will be mostly idle. For clarity, the figure illustrates the problem:



But everything could have been done like this:



The picture was a bit confusing, but the point is that we simply connect the INTx lines here with PIRQy by round-robin (PIRQA, PIRQB, PIRQC, PIRQD, PIRQA, PIRQB, PIRQC, PIRQD, PIRQA, PIRQB, PIRQC, PIRQD,. ..)

It should be noted that here it is necessary to take into account not only the fact that the same number of PCI functions are loaded on each PIRQ line. After all, some functions can create interrupts very rarely, and some constantly (for example, an Ethernet controller). In this case, even the allocation of a separate PIRQ line for interruptions from such a function can be quite justified.

Based on the foregoing, the BIOS developer, among other things, has the task of making the PIRQ lines evenly loaded with interrupts.

What should the BIOS do?


Systematized in the picture:




PIRQy signals go on the IRQz line of the selected interrupt controller (APIC / PIC). Since we want to support all possible methods of loading (see part 2 ), we need to fill in both mappings:


The last fourth point is necessary to help the OS in determining the routing interrupts. These registers are usually not used by the device itself.


It should be noted that we do not touch the $ PIR / MPtable / ACPI tables yet and consider how to configure the registers of the chipset in terms of routing interrupts before transferring control to the system loader. Interrupt tables are a topic for a separate article (possibly a future one).

So, the theoretical foundations have been studied, let's finally start practicing!

Practice


As an example, for articles in this series I use a customized motherboard with an Intel Haswell i7 processor and a LynxPoint-LP chipset. On this board, I launched the coreboot in conjunction with SeaBIOS. Coreboot provides hardware-dependent initialization, and the load (payload) SeaBIOS for it provides a BIOS interface for operating systems. In this article, I will not describe the process of configuring coreboot, but will only try to show with an example what kind of BIOS settings should I make in the chipset for routing IRQ interrupts from external devices.

Since the coreboot project is being actively developed so that the article is always relevant, we will consider the code using the example of the latest fixed version 4.9 (release 2018-12-20).

The closest motherboard to mine is the Google Beltino with the Panther variation. The main folder for this motherboard is the folder “src \ mainboard \ google \ beltino” . All settings and code specific to this board are concentrated here.

So, let's begin to understand where the above items are configured:

1) Specify which line of INTx # pulls each of the functions of PCI devices


This information is defined in the file “src / mainboard / google / beltino / romstage.c” in the rcba_config structure via the DxxIP (Device xx Interrupt Pin Register (IP)) registers. This register shows which pin INTx # (A / B / C / D) each of the functions of the device displays an interrupt.

Possible options (see the file “src / southbridge / intel / lynxpoint / pch.h” ):

0h = No interrupt 1h = INTA# 2h = INTB# 3h = INTC# 4h = INTD# 

It is assumed that several functions use the same pin.

It is assumed that functions may not use a pin for interrupts (No interrupt).
Everything as we saw in the figure at the beginning of the article.

The full code is responsible for the item indicated by us:

 /* Device interrupt pin register (board specific) */ RCBA_SET_REG_32(D31IP, (INTC << D31IP_TTIP) | (NOINT << D31IP_SIP2) | (INTB << D31IP_SMIP) | (INTA << D31IP_SIP)), RCBA_SET_REG_32(D29IP, (INTA << D29IP_E1P)), RCBA_SET_REG_32(D28IP, (INTA << D28IP_P1IP) | (INTC << D28IP_P3IP) | (INTB << D28IP_P4IP)), RCBA_SET_REG_32(D27IP, (INTA << D27IP_ZIP)), RCBA_SET_REG_32(D26IP, (INTA << D26IP_E2P)), RCBA_SET_REG_32(D22IP, (NOINT << D22IP_MEI1IP)), RCBA_SET_REG_32(D20IP, (INTA << D20IP_XHCI)), 

For a better understanding, consider a few examples:

Example 1:

In the device 0x1d (29 decimal) one function (EHCI controller).

In this case, assign an interrupt to INTA #.

00: 1d.0 - INTA #

 RCBA_SET_REG_32(D29IP, (INTA << D29IP_E1P)), 

Example 2:
The device 0x1f (31 in the decimal system) has the functions Thermal Sensor Controller (00: 1f.6), SATA controller 2 (00: 1f.2), SMBus controller (00: 1f.3), SATA controller 1 (00: 1f .2). We want to use only SMBus controller, SATA controller 1 and Thermal Sensor controller.

00: 1f.2 - INTA # (SATA controller 1)
00: 1f.3 - INTB # (SMBus controller)
00: 1f.2 - No interrupt (SATA controller 2 is not used)
00: 1f.6 - INTC # (Thermal Sensor Controller)

For such a configuration should write:

 RCBA_SET_REG_32(D31IP, (INTC << D31IP_TTIP) | (NOINT << D31IP_SIP2) | (INTB << D31IP_SMIP) | (INTA << D31IP_SIP)), 

Example 3:

In one Device, the number of functions we need is more than 4. In the 0x1c device, each function is responsible for the PCI Express port. In order for ports 0-5 to work, and the interrupts are evenly distributed between the lines, you can make the following setting:

00: 1c.0 - INTA # (PCI Express Port 0)
00.1c.1 - INTB # (PCI Express Port 1)
00.1c.2 - INTC # (PCI Express Port 2)
00.1c.3 - INTD # (PCI Express Port 3)
00.1c.4 - INTA # (PCI Express Port 4)
00.1c.5 - INTB # (PCI Express Port 5)
00.1c.6 - No interrupt (port not used)
00.1c.7 - No interrupt (port not used)

 RCBA_SET_REG_32(D28IP, (INTA << D28IP_P1IP) | (INTB << D28IP_P2IP) | (INTC << D28IP_P3IP) | (INTD << D28IP_P4IP) | (INTA << D28IP_P5IP) | (INTB << D28IP_P6IP) | (NOINT << D28IP_P7IP) | (NOINT << D28IP_P8IP)), 

2) Configure the mapping INTx # → PIRQy for each of the PCI devices


This information is also defined in the file “src \ mainboard \ google \ beltino \ romstage.c”
in the rcba_config structure, but through the DxxIR (Device xx Interrupt Route Register) registers.

The information in this register shows to which PIRQx line (A / B / C / D / E / F / G / H) each of the INTx # interrupt lines is connected.

 /* Device interrupt route registers */ RCBA_SET_REG_32(D31IR, DIR_ROUTE(PIRQG, PIRQC, PIRQB, PIRQA)),/* LPC */ RCBA_SET_REG_32(D29IR, DIR_ROUTE(PIRQD, PIRQD, PIRQD, PIRQD)),/* EHCI */ RCBA_SET_REG_32(D28IR, DIR_ROUTE(PIRQA, PIRQB, PIRQC, PIRQD)),/* PCIE */ RCBA_SET_REG_32(D27IR, DIR_ROUTE(PIRQG, PIRQG, PIRQG, PIRQG)),/* HDA */ RCBA_SET_REG_32(D22IR, DIR_ROUTE(PIRQA, PIRQA, PIRQA, PIRQA)),/* ME */ RCBA_SET_REG_32(D21IR, DIR_ROUTE(PIRQE, PIRQF, PIRQF, PIRQF)),/* SIO */ RCBA_SET_REG_32(D20IR, DIR_ROUTE(PIRQC, PIRQC, PIRQC, PIRQC)),/* XHCI */ RCBA_SET_REG_32(D23IR, DIR_ROUTE(PIRQH, PIRQH, PIRQH, PIRQH)),/* SDIO */ 

Example 1:

The device 0x1c (28 in decimal) are PCIe ports, as we have already seen.

We produce a “direct” connection:


 RCBA_SET_REG_32(D28IR, DIR_ROUTE(PIRQA, PIRQB, PIRQC, PIRQD)) 

Example 2:

Device 0x1d (29 in decimal) - one function (EHCI controller) on INTA #, the rest of the lines are not used.

Connect the INTA # line with the PIRQD:

 RCBA_SET_REG_32(D29IR, DIR_ROUTE(PIRQD, PIRQD, PIRQD, PIRQD)) 

In this case, only the first PIRQD entry (for INTA #) makes sense, the others do not make sense.

3a) Fill in the mapping PIRQy → IRQz1 (PIR → APIC)


As we have said, the mapping is often fixed here, and this case is no exception.


3b) Fill in the mapping PIRQy → IRQz2 (PIR → PIC)


In coreboot, the content for filling these registers is defined in the devicetree.cb file in the motherboard folder "src \ mainboard \ google \ beltino \".

devicetree.cb (the name devicetree is for communication with a similar concept in the Linux kernel, and “cb” is an abbreviation of coreboot) is a special file that reflects the configuration of this motherboard: which processor, chipset are used, which devices are on them, which off, etc. In addition, this file may also specify specific information for the configuration of the chipset. This is exactly the case we need:

 register "pirqa_routing" = "0x8b" register "pirqb_routing" = "0x8a" register "pirqc_routing" = "0x8b" register "pirqd_routing" = "0x8b" register "pirqe_routing" = "0x80" register "pirqf_routing" = "0x80" register "pirqg_routing" = "0x80" register "pirqh_routing" = "0x80" 

These lines set the mapping PIRQy → IRQz2. In the code after parsing the devicetree.cb file, they are transformed into the variables “config-> pirqX_routing”.

The variable “config-> pirqa_routing = 0x8b” will mean that the PIRQA is connected to the IRQ11 interrupt line (0x0b = 11) of the PIC controller, but the most severe bit (which is 0x80) means that the interrupt routing is not performed. Honestly speaking in my experience, this is a mistake, by default it is necessary to enable routing to the PIC, the operating system itself will be able to switch to I / O APIC by setting this bit to 1 if needed.

That is, in this case it would be better to write:

 register "pirqa_routing" = "0x0b" register "pirqb_routing" = "0x0a" register "pirqc_routing" = "0x0b" register "pirqd_routing" = "0x0b" register "pirqe_routing" = "0x80" # not used register "pirqf_routing" = "0x80" # not used register "pirqg_routing" = "0x80" # not used register "pirqh_routing" = "0x80" # not used 

We did not enable the last 4 interrupts, since IRQ0 interrupt is always used under the system timer and is clearly inaccessible (see General IBM-PC Compatible Interrupt Information ).

But if we take a closer look at point 2), we will see that some PCI devices use PIRQE-PIRQH lines, so leave them unconnected the right path to non-working devices.

So it is better to write something like this:
 register "pirqa_routing" = "0x03" register "pirqb_routing" = "0x04" register "pirqc_routing" = "0x05" register "pirqd_routing" = "0x06" register "pirqe_routing" = "0x0a" register "pirqf_routing" = "0x0b" register "pirqg_routing" = "0x0e" register "pirqh_routing" = "0x0f" 


The actual filling of the corresponding registers takes place in the src \ southbridge \ intel \ lynxpoint \ lpc.c file in the pch_pirq_init function.

Fragment of code responsible for filling registers:

 /* Get the chip configuration */ config_t *config = dev->chip_info; pci_write_config8(dev, PIRQA_ROUT, config->pirqa_routing); pci_write_config8(dev, PIRQB_ROUT, config->pirqb_routing); pci_write_config8(dev, PIRQC_ROUT, config->pirqc_routing); pci_write_config8(dev, PIRQD_ROUT, config->pirqd_routing); pci_write_config8(dev, PIRQE_ROUT, config->pirqe_routing); pci_write_config8(dev, PIRQF_ROUT, config->pirqf_routing); pci_write_config8(dev, PIRQG_ROUT, config->pirqg_routing); pci_write_config8(dev, PIRQH_ROUT, config->pirqh_routing); 

Register address constants are described in the same pch.h file .

 #define PIRQA_ROUT 0x60 #define PIRQB_ROUT 0x61 #define PIRQC_ROUT 0x62 #define PIRQD_ROUT 0x63 #define PIRQE_ROUT 0x68 #define PIRQF_ROUT 0x69 #define PIRQG_ROUT 0x6A #define PIRQH_ROUT 0x6B 

Mapping PIRQy → IRQz2 for this chipset is written to the PCI LPC device (address 00: 1f.0) in the PIRQy_ROUT registers. It should be noted that often not all 15 IRQz2 lines on the PIC are allowed for use, but only a part (for example, 3,4,5,6,7,9,10,11,12,14,15). In the description of these registers there should be information about which IRQs are available for assigning interrupts from PIRQ lines to them. So the mapping proposed by us is possible only if PIRQ assignment on the IRQ3, IRQ4, IRQ5, IRQ6, IRQ10, IRQ11, IRQ14, IRQ15 lines is available. But if we carefully look at the comment before the pch_pirq_init function, we will see that it is:

 /* PIRQ[n]_ROUT[3:0] - PIRQ Routing Control * 0x00 - 0000 = Reserved * 0x01 - 0001 = Reserved * 0x02 - 0010 = Reserved * 0x03 - 0011 = IRQ3 * 0x04 - 0100 = IRQ4 * 0x05 - 0101 = IRQ5 * 0x06 - 0110 = IRQ6 * 0x07 - 0111 = IRQ7 * 0x08 - 1000 = Reserved * 0x09 - 1001 = IRQ9 * 0x0A - 1010 = IRQ10 * 0x0B - 1011 = IRQ11 * 0x0C - 1100 = IRQ12 * 0x0D - 1101 = Reserved * 0x0E - 1110 = IRQ14 * 0x0F - 1111 = IRQ15 * PIRQ[n]_ROUT[7] - PIRQ Routing Control * 0x80 - The PIRQ is not routed. */ 

4) Fill in the Interrupt Line / Interrupt Pin registers for each PCI function


In the PCI configuration space (each PCI function has a standard) there are 2 registers of interest to us:


Let's start with the last one. The Interrupt Pin register will be filled automatically based on the chipset settings (DxxIP registers) made by us in step 1 and will be Read-Only.

So it remains only to fill the Interrupt Line register with an IRQz2 interrupt for each PCI function.

Knowing the mapping PIRQy → IRQz2 (item 3b), and mapping INTx # → PIRQy (item 2), you can easily fill the Interrupt Line register for each function, knowing what kind of INTx # interrupt it uses (item 1).

In coreboot, the Interrupt Line registers are also filled in the src \ southbridge \ intel \ lynxpoint \ lpc.c file in the pch_pirq_init function:

 /* Eric Biederman once said we should let the OS do this. * I am not so sure anymore he was right. */ for (irq_dev = all_devices; irq_dev; irq_dev = irq_dev->next) { u8 int_pin=0, int_line=0; if (!irq_dev->enabled || irq_dev->path.type != DEVICE_PATH_PCI) continue; int_pin = pci_read_config8(irq_dev, PCI_INTERRUPT_PIN); switch (int_pin) { case 1: /* INTA# */ int_line = config->pirqa_routing; break; case 2: /* INTB# */ int_line = config->pirqb_routing; break; case 3: /* INTC# */ int_line = config->pirqc_routing; break; case 4: /* INTD# */ int_line = config->pirqd_routing; break; } if (!int_line) continue; pci_write_config8(irq_dev, PCI_INTERRUPT_LINE, int_line); } 

For some reason, this code assumes that the mapping is anyway INTA # → PIRQA, INTB # → PIRQB, INTC # → PIRQC, INTD # → PIRQD. Although, in fact, we have seen that it may be different (see point 2).

In general, "Eric Biederman once said," and we scattered everything:

 $ grep "Eric Biederman once said" -r src/ src/southbridge/intel/fsp_bd82x6x/lpc.c: /* Eric Biederman once said we should let the OS do this. src/southbridge/intel/i82801gx/lpc.c: /* Eric Biederman once said we should let the OS do this. src/southbridge/intel/i82801ix/lpc.c: /* Eric Biederman once said we should let the OS do this. src/southbridge/intel/lynxpoint/lpc.c: /* Eric Biederman once said we should let the OS do this. src/southbridge/intel/sch/lpc.c: /* Eric Biederman once said we should let the OS do this. 

In general, the coreboot does not care much about supporting legacy interrupt modes. So you shouldn't be surprised at such an error. When booting a modern OS, this will not hurt you, but if you suddenly need to boot Linux with the “acpi = off nolapic” options, then this is unlikely to be done.

Conclusion


In conclusion, we will repeat the typical information that must be configured in the chipset for routing PCI interrupts:

  1. Specify which line of INTx # pulls each of the functions of PCI devices
  2. Configure INTx # → PIRQy mapping for each PCI device
  3. Fill in the PIRQy → IRQz1 mapping (PIR → APIC) and the PIRQy → IRQz2 mapping (PIR → PIC)
  4. Fill in the Interrupt Line / Interrupt Pin registers of the PCI configuration space for each PCI function.

Source: https://habr.com/ru/post/440304/