8174c430e4
Implement get_user_pages_fast without locking in the fastpath on x86. Do an optimistic lockless pagetable walk, without taking mmap_sem or any page table locks or even mmap_sem. Page table existence is guaranteed by turning interrupts off (combined with the fact that we're always looking up the current mm, means we can do the lockless page table walk within the constraints of the TLB shootdown design). Basically we can do this lockless pagetable walk in a similar manner to the way the CPU's pagetable walker does not have to take any locks to find present ptes. This patch (combined with the subsequent ones to convert direct IO to use it) was found to give about 10% performance improvement on a 2 socket 8 core Intel Xeon system running an OLTP workload on DB2 v9.5 "To test the effects of the patch, an OLTP workload was run on an IBM x3850 M2 server with 2 processors (quad-core Intel Xeon processors at 2.93 GHz) using IBM DB2 v9.5 running Linux 2.6.24rc7 kernel. Comparing runs with and without the patch resulted in an overall performance benefit of ~9.8%. Correspondingly, oprofiles showed that samples from __up_read and __down_read routines that is seen during thread contention for system resources was reduced from 2.8% down to .05%. Monitoring the /proc/vmstat output from the patched run showed that the counter for fast_gup contained a very high number while the fast_gup_slow value was zero." (fast_gup is the old name for get_user_pages_fast, fast_gup_slow is a counter we had for the number of times the slowpath was invoked). The main reason for the improvement is that DB2 has multiple threads each issuing direct-IO. Direct-IO uses get_user_pages, and thus the threads contend the mmap_sem cacheline, and can also contend on page table locks. I would anticipate larger performance gains on larger systems, however I think DB2 uses an adaptive mix of threads and processes, so it could be that thread contention remains pretty constant as machine size increases. In which case, we stuck with "only" a 10% gain. The downside of using get_user_pages_fast is that if there is not a pte with the correct permissions for the access, we end up falling back to get_user_pages and so the get_user_pages_fast is a bit of extra work. However this should not be the common case in most performance critical code. [akpm@linux-foundation.org: coding-style fixes] [akpm@linux-foundation.org: build fix] [akpm@linux-foundation.org: Kconfig fix] [akpm@linux-foundation.org: Makefile fix/cleanup] [akpm@linux-foundation.org: warning fix] Signed-off-by: Nick Piggin <npiggin@suse.de> Cc: Dave Kleikamp <shaggy@austin.ibm.com> Cc: Andy Whitcroft <apw@shadowen.org> Cc: Ingo Molnar <mingo@elte.hu> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Andi Kleen <andi@firstfloor.org> Cc: Dave Kleikamp <shaggy@austin.ibm.com> Cc: Badari Pulavarty <pbadari@us.ibm.com> Cc: Zach Brown <zach.brown@oracle.com> Cc: Jens Axboe <jens.axboe@oracle.com> Reviewed-by: Peter Zijlstra <a.p.zijlstra@chello.nl> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
26 lines
709 B
Makefile
26 lines
709 B
Makefile
obj-y := init_$(BITS).o fault.o ioremap.o extable.o pageattr.o mmap.o \
|
|
pat.o pgtable.o
|
|
|
|
obj-$(CONFIG_HAVE_GET_USER_PAGES_FAST) += gup.o
|
|
obj-$(CONFIG_X86_32) += pgtable_32.o
|
|
|
|
obj-$(CONFIG_HUGETLB_PAGE) += hugetlbpage.o
|
|
obj-$(CONFIG_X86_PTDUMP) += dump_pagetables.o
|
|
|
|
obj-$(CONFIG_HIGHMEM) += highmem_32.o
|
|
|
|
obj-$(CONFIG_MMIOTRACE_HOOKS) += kmmio.o
|
|
obj-$(CONFIG_MMIOTRACE) += mmiotrace.o
|
|
mmiotrace-y := pf_in.o mmio-mod.o
|
|
obj-$(CONFIG_MMIOTRACE_TEST) += testmmiotrace.o
|
|
|
|
ifeq ($(CONFIG_X86_32),y)
|
|
obj-$(CONFIG_NUMA) += discontig_32.o
|
|
else
|
|
obj-$(CONFIG_NUMA) += numa_64.o
|
|
obj-$(CONFIG_K8_NUMA) += k8topology_64.o
|
|
endif
|
|
obj-$(CONFIG_ACPI_NUMA) += srat_$(BITS).o
|
|
|
|
obj-$(CONFIG_MEMTEST) += memtest.o
|