android_kernel_xiaomi_sm8350/arch/x86
Ingo Molnar 4aae070252 x86: fix "Kernel panic - not syncing: IO-APIC + timer doesn't work!"
this is the tale of a full day spent debugging an ancient but elusive bug.

after booting up thousands of random .config kernels, i finally happened
to generate a .config that produced the following rare bootup failure
on 32-bit x86:

| ..TIMER: vector=0x31 apic1=0 pin1=2 apic2=-1 pin2=-1
| ..MP-BIOS bug: 8254 timer not connected to IO-APIC
| ...trying to set up timer (IRQ0) through the 8259A ...  failed.
| ...trying to set up timer as Virtual Wire IRQ... failed.
| ...trying to set up timer as ExtINT IRQ... failed :(.
| Kernel panic - not syncing: IO-APIC + timer doesn't work!  Boot with apic=debug
| and send a report.  Then try booting with the 'noapic' option

this bug has been reported many times during the years, but it was never
reproduced nor fixed.

the bug that i hit was extremely sensitive to .config details.

First i did a .config-bisection - suspecting some .config detail.
That led to CONFIG_X86_MCE: enabling X86_MCE magically made the bug disappear
and the system would boot up just fine.

Debugging my way through the MCE code ended up identifying two unlikely
candidates: the thing that made a real difference to the hang was that
X86_MCE did two printks:

 Intel machine check architecture supported.
 Intel machine check reporting enabled on CPU#1.

Adding the same printks to a !CONFIG_X86_MCE kernel made the bug go away!

this left timing as the main suspect: i experimented with adding various
udelay()s to the arch/x86/kernel/io_apic_32.c:check_timer() function, and
the race window turned out to be narrower than 30 microseconds (!).

That made debugging especially funny, debugging without having printk
ability before the bug hits is ... interesting ;-)

eventually i started suspecting IRQ activities - those are pretty much the
only thing that happen this early during bootup and have the timescale of
a few dozen microseconds. Also, check_timer() changes the IRQ hardware
in various creative ways, so the main candidate became IRQ0 interaction.

i've added a counter to track timer irqs (on which core they arrived, at
what exact time, etc.) and found that no timer IRQ would arrive after the
bug condition hits - even if we re-enable IRQ0 and re-initialize the i8259A,
but that we'd get a small number of timer irqs right around the time when we
call the check_timer() function.

Eventually i got the following backtrace triggered from debug code in the
timer interrupt:

...trying to set up timer as Virtual Wire IRQ... failed.
...trying to set up timer as ExtINT IRQ...
Pid: 1, comm: swapper Not tainted (2.6.24-rc5 #57)
EIP: 0060:[<c044d57e>] EFLAGS: 00000246 CPU: 0
EIP is at _spin_unlock_irqrestore+0x5/0x1c
EAX: c0634178 EBX: 00000000 ECX: c4947d63 EDX: 00000246
ESI: 00000002 EDI: 00010031 EBP: c04e0f2e ESP: f7c41df4
 DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
 CR0: 8005003b CR2: ffe04000 CR3: 00630000 CR4: 000006d0
 DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
 DR6: ffff0ff0 DR7: 00000400
  [<c05f5784>] setup_IO_APIC+0x9c3/0xc5c

the spin_unlock() was called from init_8259A(). Wait ... we have an IRQ0
entry while we are in the middle of setting up the local APIC, the i8259A
and the PIT??

That is certainly not how it's supposed to work! check_timer() was supposed
to be called with irqs turned off - but this eroded away sometime in the
past. This code would still work most of the time because this code runs
very quickly, but just the right timing conditions are present and IRQ0
hits in this small, ~30 usecs window, timer irqs stop and the system does
not boot up. Also, given how early this is during bootup, the hang is
very deterministic - but it would only occur on certain machines (and
certain configs).

The fix was quite simple: disable/restore interrupts properly in this
function. With that in place the test-system now boots up just fine.

(64-bit x86 io_apic_64.c had the same bug.)

Phew! One down, only 1500 other kernel bugs are left ;-)

Signed-off-by: Ingo Molnar <mingo@elte.hu>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
2007-12-18 18:05:58 +01:00
..
boot x86 setup: don't recalculate ss:esp unless really necessary 2007-11-28 18:17:17 -08:00
configs x86 gart: rename CONFIG_IOMMU to CONFIG_GART_IOMMU 2007-10-30 00:22:22 +01:00
crypto x86: merge arch/x86/crypto Makefiles 2007-10-23 22:37:23 +02:00
ia32 x86 - 32-bit ptrace emulation mishandles 6th arg 2007-11-10 04:30:36 +01:00
kernel x86: fix "Kernel panic - not syncing: IO-APIC + timer doesn't work!" 2007-12-18 18:05:58 +01:00
lguest lguest: prevent VISWS or VOYAGER randconfigs 2007-11-29 09:24:55 -08:00
lib x86: disable preemption in delay_tsc() 2007-11-14 18:45:44 -08:00
mach-default spelling fixes: arch/i386/ 2007-10-20 01:13:56 +02:00
mach-es7000 i386: es7000 minor cleanups 2007-10-17 20:16:15 +02:00
mach-generic spelling fixes: arch/i386/ 2007-10-20 01:13:56 +02:00
mach-visws [x86] remove uses of magic macros for boot_params access 2007-10-16 17:38:31 -07:00
mach-voyager x86: fix smp init sections 2007-11-17 16:27:00 +01:00
math-emu kbuild: fix up CFLAGS usage 2007-10-14 21:49:42 +02:00
mm memory hotplug x86_64: fix section mismatch in init_memory_mapping() 2007-11-29 09:24:54 -08:00
oprofile oprofile: op_model_athlon.c support for AMD family 10h barcelona performance counters 2007-12-18 18:05:58 +01:00
pci Merge git://git.kernel.org/pub/scm/linux/kernel/git/x86/linux-2.6-x86 2007-11-26 19:41:28 -08:00
power
vdso x86: ignore the sys_getcpu() tcache parameter 2007-11-17 16:27:00 +01:00
video
xen xen: relax signature check 2007-12-10 19:46:58 -08:00
Kconfig Tiny clean-up of OPROFILE/KPROBES configuration 2007-12-06 09:41:12 -08:00
Kconfig.cpu x86: arch/x86/Kconfig.cpu unification 2007-11-12 21:02:19 +01:00
Kconfig.debug remove the dead X86_REMOTE_DEBUG option 2007-10-30 00:22:22 +01:00
Makefile x86: correctly set UTS_MACHINE for "make ARCH=x86" 2007-11-26 17:38:53 -08:00
Makefile_32 x86: do not use $(ARCH) when not needed 2007-11-12 21:02:20 +01:00
Makefile_32.cpu x86: move i386 and x86_64 Makefiles to arch/x86 2007-10-25 22:27:34 +02:00
Makefile_64 x86: do not use $(ARCH) when not needed 2007-11-12 21:02:20 +01:00