Climbing Elbrus - Reconnaissance in battle. Technical Part 2. Interrupts, exceptions, system timer
We continue to explore Elbrus by porting Embox to it . This article is the second part of a technical article on Elbrus architecture. The first part dealt with stacks, registers, and so on. Before reading this part, we recommend that you study the first, as it talks about the basic things in Elbrus architecture. This section will focus on timers, interrupts, and exceptions. This, again, is not official documentation. For it, you should contact the developers of Elbrus in the MCST .
Getting to the study of Elbrus, we wanted to quickly start the timer, because, as you know, preemptive multitasking does not work without it. To do this, it seemed enough to implement the interrupt controller and the timer itself, but we ran into
The code is as follows:
#define UPSR_DI (1 << 3) /* Определен в .h файле */
rrs %upsr, %r1
ors %r1, UPSR_DI, %r1 /* upsr |= UPSR_DI; */
rws %r1, %upsr
vfdi /* Вот здесь должно выработаться исключение */Launched. But nothing happened, the system hung somewhere, nothing was output to the console. Actually, we saw this when we tried to start the interrupt from the timer, but then there were many components, and here it was clear that something interrupted the sequential progress of our program, and control was transferred to the exception table (in terms of Elbrus architecture it’s more correct to talk not about the table interruptions and about the table of exceptions). We assumed that the processor still threw an exception, but there was some kind of “garbage” where it transferred control. As it turned out, he transfers control to the very place where we put the Embox image, which means that there was an entry point - the entry function.
For verification, we did the following. Started a counter of entries in entry (). Initially, all CPUs start with interrupts turned off, go into entry (), after which we leave only one core active, all the rest go into an endless loop. After the counter is equal to the number of CPUs, we consider that all subsequent hits in entry are exceptions. I remind you that before it was as described in our very first article about Elbrus
cpuid = __e2k_atomic32_add(1, &last_cpuid);
if (cpuid > 1) {
/* XXX currently we support only single core */
while(1);
}
/* copy of trap table */
memcpy((void*)0, &_t_entry, 0x1800);
kernel_start();Did so
/* Since we enable exceptions only when all CPUs except the main one
* reached the idle state (cpu_idle), we can rely that order and can
* guarantee exceptions happen strictly after all CPUS entries. */
if (entries_count >= CPU_COUNT) {
/* Entering here because of expection or interrupt */
e2k_trap_handler(regs);
...
}
/* It wasn't exception, so we decide this usual program execution,
* that is, Embox started on CPU0 or CPU1 */
e2k_wait_all();
entries_count = __e2k_atomic32_add(1, &entries_count);
if (entries_count > 1) {
/* XXX currently we support only single core */
cpu_idle();
}
e2k_kernel_start();
}And finally we saw the reaction to entering the interrupt (just with the help of printf we printed a line).
Here it’s worth explaining that initially in the first version we expected to copy the exception table, but firstly, it turned out that it was at our address, and secondly, we were not able to make the correct copy. I had to rewrite linker scripts, the entry point into the system, and the interrupt handler, that is, I needed the assembler part, about it a little later.
This is how the part of the modified part of the script linker now looks:
.text : {
_start = .;
_t_entry = .;
/* Interrupt handler */
*(.ttable_entry0)
. = _t_entry + 0x800;
/* Syscall handler */
*(.ttable_entry1)
. = _t_entry + 0x1000;
/* longjmp handler */
*(.ttable_entry2)
. = _t_entry + 0x1800;
_t_entry_end = .;
*(.e2k_entry)
*(.cpu_idle)
/* text */
}
that is, we removed the entry section for the exception table. The cpu_idle section is also located there for those CPUs that are not used.
This is what the input function looks like for our active kernel, on which Embox will run:
static void e2k_kernel_start(void) {
extern void kernel_start(void);
int psr;
/* Ждем пока остальные CPU “уснут” */
while (idled_cpus_count < CPU_COUNT - 1)
;
...
/* Отключаем операции с плавающей точкой, они разрешены по умолчанию */
e2k_upsr_write(e2k_upsr_read() & ~UPSR_FE);
kernel_start(); /* Входим в Embox */
}Well, according to the VFDI instruction, an exception was thrown. Now you need to get his number to make sure that this is the correct exception. For this, in Elbrus there are TIR interrupt information registers (Trap Info registers). They contain information about the last few commands, that is, the final part of the trace. Trace gathers during program execution and freezes when it enters an interrupt. TIR includes the low (64 bits) and high (64 bits) parts. The low word contains the exception flags, and the high word contains a pointer to the instruction that led to the exception and the current TIR number. Accordingly, in our case, exc_d_interrupt is the 4th bit.
Note We still have some misunderstanding regarding the depth (number) of TIRs. The documentation provides:
“The TIR memory depth, that is, the number of Trap Info registers, is determined by theIn practice, we see the depth = 1, and therefore we use only the TIR0 register.
TIR_NUM macro, equal to the number of stages of the processor pipeline required to
issue all possible special situations. TIR_NUM = 19; ”
Specialists at the MCST explained to us that everything is correct, and there will only be TIR0 for “accurate” interrupts, and for other situations there may be something else. But since while we are only talking about timer interrupts, this does not bother us.
Ok, now we will examine what is needed to correctly enter / exit the exception handler. Actually it is necessary to save at the input and restore the following 5 registers at the output. Three control transfer preparation registers are ctpr [1,2,3], and two cycle control registers are ILCR (Register of initial values of the cycle counter) and LSR (Register of cycle status).
.type ttable_entry0,@function
ttable_entry0:
setwd wsz = 0x10, nfx = 1;
rrd %ctpr1, %dr1
rrd %ctpr2, %dr2
rrd %ctpr3, %dr3
rrd %ilcr, %dr4
rrd %lsr, %dr5
/* sizeof pt_regs */
getsp -(5 * 8), %dr0
std %dr1, [%dr0 + PT_CTRP1] /* regs->ctpr1 = ctpr1 */
std %dr2, [%dr0 + PT_CTRP2] /* regs->ctpr2 = ctpr2 */
std %dr3, [%dr0 + PT_CTRP3] /* regs->ctpr3 = ctpr3 */
std %dr4, [%dr0 + PT_ILCR] /* regs->ilcr = ilcr */
std %dr5, [%dr0 + PT_LSR] /* regs->lsr = lsr */
disp %ctpr1, e2k_entry
ct %ctpr1Actually, that’s all, after exiting the exception handler, you need to restore these 5 registers.
We do this with a macro:
#define RESTORE_COMMON_REGS(regs) \
({ \
uint64_t ctpr1 = regs->ctpr1, ctpr2 = regs->ctpr2, \
ctpr3 = regs->ctpr3, lsr = regs->lsr, \
ilcr = regs->ilcr; \
/* ctpr2 is restored first because of tight time constraints \
* on restoring ctpr2 and aaldv. */ \
E2K_SET_DSREG(ctpr1, ctpr1); \
E2K_SET_DSREG(ctpr2, ctpr2); \
E2K_SET_DSREG(ctpr3, ctpr3); \
E2K_SET_DSREG(lsr, lsr); \
E2K_SET_DSREG(ilcr, ilcr); \
})It is also important not to forget after the restoration of the registers to call the DONE operation (Return from the hardware interrupt handler). This operation is needed, in particular, in order to correctly process the interrupted control transfer operations. We do this with a macro:
#define E2K_DONE \
do { \
asm volatile ("{nop 3} {done}" ::: "ctpr3"); \
} while (0)Actually, we do the return from the interrupt directly in C code using these two macros.
/* Entering here because of expection or interrupt */
e2k_trap_handler(regs);
RESTORE_COMMON_REGS(regs);
E2K_DONE;External interrupts
Let's start with how to enable external interrupts. In Elbrus, APIC (or rather its analog) is used as an interrupt controller; Embox already had this driver. Therefore, it was possible to pick up a system timer for it. There are two timers, one of which is very similar to PIT , the other LAPIC Timer , also quite standard, so it makes no sense to talk about them. Both that and that looked simple, and that and that already existed in Embox, but the driver of the LAPIC timer looked more perspective, besides implementation of the PIT timer seemed to us more non-standard. Therefore, it seemed easier to complete. In addition, the official documentation described the registers APIC and LAPIC, which were slightly different from the originals. Bringing them makes no sense, as you can see in the original.
In addition to allowing interrupts in APIC, you must enable interrupt handling through the PSR / UPSR registers. Both registers have flags allowing external interrupts and non-maskable interrupts. BUT here it is very important to note that the PSR register is local to the function (this was discussed in the first technical part ). And this means that if you set it inside a function, then when you call all subsequent functions, it will be inherited, but when you return from the function, it will return to its original state. Hence the question, but how to manage interrupts?
We use the following solution. The PSR register allows you to enable management through UPSR, which is already global (which is what we need). Therefore, we enable control via UPSR directly (important!) Before the Embox core login function:
/* PSR is local register and makes sense only within a function,
* so we set it here before kernel start. */
asm volatile ("rrs %%psr, %0" : "=r"(psr) :);
psr |= (PSR_IE | PSR_NMIE | PSR_UIE);
asm volatile ("rws %0, %%psr" : : "ri"(psr));
kernel_start();Somehow, by chance, after refactoring, I took and put these lines into a separate function ... And the register is local to the function. It’s clear that everything has broken :)
So, in the processor, everything you need seems to be turned on, go to the interrupt controller.
As we have seen above, information about the exception number is in the TIR register. Further, the 32nd bit in this register reports that an external interrupt has occurred.
After the timer was turned on, a couple of days of torment followed, since no interruption could be obtained. The reason was amusing enough. There are 64-bit pointers in Elbrus, and the register address in APIC got into uint32_t, that's why we used them. But it turned out that if you need, for example, cast 0xF0000000 to a pointer, then you will get not 0xF0000000, but 0xFFFFFFFFF0000000. That is, the compiler will expand your unsigned int sign.
Of course, here it was necessary to use uintptr_t, since, as it turned out, in the C99 standard such casts are implementation defined.
After we finally saw the raised 32nd bit in TIR, we began to look for how to get the interrupt number. It turned out to be quite simple, although not at all like on x86, this is one of the differences between the LAPIC implementations. For Elbrus, to get the interrupt number, you need to get into the special LAPIC register:
#define APIC_VECT (0xFEE00000 + 0xFF0)where 0xFEE00000 is the base address of the LAPIC registers.
That's all, it turned out to pick up both the system timer and the LAPIC timer.
Conclusion
The information given in the first two technical parts of the article about Elbrus architecture is enough to implement hardware interrupts and preemptive multitasking in any OS. Actually, the given screenshots testify to this.
This is not the last technical part about Elbrus architecture. Now we are mastering memory management (MMU) in Elbrus, we hope to talk about it soon. We need this not only for the implementation of virtual address spaces, but also for normal work with peripherals, because through this mechanism you can disable or enable caching of a specific area of the address space.
Everything that is written in the article can be found in the Embox repository.. You can also build and run, if of course there is a hardware platform. True, a compiler is needed for this, and it can only be obtained at the MCST . Official documentation can be requested there.