0069455bcb
Patch series "Memory allocation profiling", v6.
Overview:
Low overhead [1] per-callsite memory allocation profiling. Not just for
debug kernels, overhead low enough to be deployed in production.
Example output:
root@moria-kvm:~# sort -rn /proc/allocinfo
127664128 31168 mm/page_ext.c:270 func:alloc_page_ext
56373248 4737 mm/slub.c:2259 func:alloc_slab_page
14880768 3633 mm/readahead.c:247 func:page_cache_ra_unbounded
14417920 3520 mm/mm_init.c:2530 func:alloc_large_system_hash
13377536 234 block/blk-mq.c:3421 func:blk_mq_alloc_rqs
11718656 2861 mm/filemap.c:1919 func:__filemap_get_folio
9192960 2800 kernel/fork.c:307 func:alloc_thread_stack_node
4206592 4 net/netfilter/nf_conntrack_core.c:2567 func:nf_ct_alloc_hashtable
4136960 1010 drivers/staging/ctagmod/ctagmod.c:20 [ctagmod] func:ctagmod_start
3940352 962 mm/memory.c:4214 func:alloc_anon_folio
2894464 22613 fs/kernfs/dir.c:615 func:__kernfs_new_node
...
Usage:
kconfig options:
- CONFIG_MEM_ALLOC_PROFILING
- CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT
- CONFIG_MEM_ALLOC_PROFILING_DEBUG
adds warnings for allocations that weren't accounted because of a
missing annotation
sysctl:
/proc/sys/vm/mem_profiling
Runtime info:
/proc/allocinfo
Notes:
[1]: Overhead
To measure the overhead we are comparing the following configurations:
(1) Baseline with CONFIG_MEMCG_KMEM=n
(2) Disabled by default (CONFIG_MEM_ALLOC_PROFILING=y &&
CONFIG_MEM_ALLOC_PROFILING_BY_DEFAULT=n)
(3) Enabled by default (CONFIG_MEM_ALLOC_PROFILING=y &&
CONFIG_MEM_ALLOC_PROFILING_BY_DEFAULT=y)
(4) Enabled at runtime (CONFIG_MEM_ALLOC_PROFILING=y &&
CONFIG_MEM_ALLOC_PROFILING_BY_DEFAULT=n && /proc/sys/vm/mem_profiling=1)
(5) Baseline with CONFIG_MEMCG_KMEM=y && allocating with __GFP_ACCOUNT
(6) Disabled by default (CONFIG_MEM_ALLOC_PROFILING=y &&
CONFIG_MEM_ALLOC_PROFILING_BY_DEFAULT=n) && CONFIG_MEMCG_KMEM=y
(7) Enabled by default (CONFIG_MEM_ALLOC_PROFILING=y &&
CONFIG_MEM_ALLOC_PROFILING_BY_DEFAULT=y) && CONFIG_MEMCG_KMEM=y
Performance overhead:
To evaluate performance we implemented an in-kernel test executing
multiple get_free_page/free_page and kmalloc/kfree calls with allocation
sizes growing from 8 to 240 bytes with CPU frequency set to max and CPU
affinity set to a specific CPU to minimize the noise. Below are results
from running the test on Ubuntu 22.04.2 LTS with 6.8.0-rc1 kernel on
56 core Intel Xeon:
kmalloc pgalloc
(1 baseline) 6.764s 16.902s
(2 default disabled) 6.793s (+0.43%) 17.007s (+0.62%)
(3 default enabled) 7.197s (+6.40%) 23.666s (+40.02%)
(4 runtime enabled) 7.405s (+9.48%) 23.901s (+41.41%)
(5 memcg) 13.388s (+97.94%) 48.460s (+186.71%)
(6 def disabled+memcg) 13.332s (+97.10%) 48.105s (+184.61%)
(7 def enabled+memcg) 13.446s (+98.78%) 54.963s (+225.18%)
Memory overhead:
Kernel size:
text data bss dec diff
(1) 26515311 18890222 17018880 62424413
(2) 26524728 19423818 16740352 62688898 264485
(3) 26524724 19423818 16740352 62688894 264481
(4) 26524728 19423818 16740352 62688898 264485
(5) 26541782 18964374 16957440 62463596 39183
Memory consumption on a 56 core Intel CPU with 125GB of memory:
Code tags: 192 kB
PageExts: 262144 kB (256MB)
SlabExts: 9876 kB (9.6MB)
PcpuExts: 512 kB (0.5MB)
Total overhead is 0.2% of total memory.
Benchmarks:
Hackbench tests run 100 times:
hackbench -s 512 -l 200 -g 15 -f 25 -P
baseline disabled profiling enabled profiling
avg 0.3543 0.3559 (+0.0016) 0.3566 (+0.0023)
stdev 0.0137 0.0188 0.0077
hackbench -l 10000
baseline disabled profiling enabled profiling
avg 6.4218 6.4306 (+0.0088) 6.5077 (+0.0859)
stdev 0.0933 0.0286 0.0489
stress-ng tests:
stress-ng --class memory --seq 4 -t 60
stress-ng --class cpu --seq 4 -t 60
Results posted at: https://evilpiepirate.org/~kent/memalloc_prof_v4_stress-ng/
[2] https://lore.kernel.org/all/20240306182440.2003814-1-surenb@google.com/
This patch (of 37):
The next patch drops vmalloc.h from a system header in order to fix a
circular dependency; this adds it to all the files that were pulling it in
implicitly.
[kent.overstreet@linux.dev: fix arch/alpha/lib/memcpy.c]
Link: https://lkml.kernel.org/r/20240327002152.3339937-1-kent.overstreet@linux.dev
[surenb@google.com: fix arch/x86/mm/numa_32.c]
Link: https://lkml.kernel.org/r/20240402180933.1663992-1-surenb@google.com
[kent.overstreet@linux.dev: a few places were depending on sizes.h]
Link: https://lkml.kernel.org/r/20240404034744.1664840-1-kent.overstreet@linux.dev
[arnd@arndb.de: fix mm/kasan/hw_tags.c]
Link: https://lkml.kernel.org/r/20240404124435.3121534-1-arnd@kernel.org
[surenb@google.com: fix arc build]
Link: https://lkml.kernel.org/r/20240405225115.431056-1-surenb@google.com
Link: https://lkml.kernel.org/r/20240321163705.3067592-1-surenb@google.com
Link: https://lkml.kernel.org/r/20240321163705.3067592-2-surenb@google.com
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
Signed-off-by: Suren Baghdasaryan <surenb@google.com>
Signed-off-by: Arnd Bergmann <arnd@arndb.de>
Reviewed-by: Pasha Tatashin <pasha.tatashin@soleen.com>
Tested-by: Kees Cook <keescook@chromium.org>
Cc: Alexander Viro <viro@zeniv.linux.org.uk>
Cc: Alex Gaynor <alex.gaynor@gmail.com>
Cc: Alice Ryhl <aliceryhl@google.com>
Cc: Andreas Hindborg <a.hindborg@samsung.com>
Cc: Benno Lossin <benno.lossin@proton.me>
Cc: "Björn Roy Baron" <bjorn3_gh@protonmail.com>
Cc: Boqun Feng <boqun.feng@gmail.com>
Cc: Christoph Lameter <cl@linux.com>
Cc: Dennis Zhou <dennis@kernel.org>
Cc: Gary Guo <gary@garyguo.net>
Cc: Miguel Ojeda <ojeda@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Tejun Heo <tj@kernel.org>
Cc: Vlastimil Babka <vbabka@suse.cz>
Cc: Wedson Almeida Filho <wedsonaf@gmail.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
105 lines
2.5 KiB
C
105 lines
2.5 KiB
C
/* SPDX-License-Identifier: GPL-2.0-only */
|
|
/*
|
|
* Copyright (C) 2004, 2007-2010, 2011-2012, 2019-20 Synopsys, Inc. (www.synopsys.com)
|
|
*
|
|
* MMUv3 (arc700) / MMUv4 (archs) are software page walked and software managed.
|
|
* This file contains the TLB access registers and commands
|
|
*/
|
|
|
|
#ifndef _ASM_ARC_MMU_ARCV2_H
|
|
#define _ASM_ARC_MMU_ARCV2_H
|
|
|
|
#include <soc/arc/aux.h>
|
|
|
|
/*
|
|
* TLB Management regs
|
|
*/
|
|
#define ARC_REG_MMU_BCR 0x06f
|
|
|
|
#ifdef CONFIG_ARC_MMU_V3
|
|
#define ARC_REG_TLBPD0 0x405
|
|
#define ARC_REG_TLBPD1 0x406
|
|
#define ARC_REG_TLBPD1HI 0 /* Dummy: allows common code */
|
|
#define ARC_REG_TLBINDEX 0x407
|
|
#define ARC_REG_TLBCOMMAND 0x408
|
|
#define ARC_REG_PID 0x409
|
|
#define ARC_REG_SCRATCH_DATA0 0x418
|
|
#else
|
|
#define ARC_REG_TLBPD0 0x460
|
|
#define ARC_REG_TLBPD1 0x461
|
|
#define ARC_REG_TLBPD1HI 0x463
|
|
#define ARC_REG_TLBINDEX 0x464
|
|
#define ARC_REG_TLBCOMMAND 0x465
|
|
#define ARC_REG_PID 0x468
|
|
#define ARC_REG_SCRATCH_DATA0 0x46c
|
|
#endif
|
|
|
|
/* Bits in MMU PID reg */
|
|
#define __TLB_ENABLE (1 << 31)
|
|
#define __PROG_ENABLE (1 << 30)
|
|
#define MMU_ENABLE (__TLB_ENABLE | __PROG_ENABLE)
|
|
|
|
/* Bits in TLB Index reg */
|
|
#define TLB_LKUP_ERR 0x80000000
|
|
|
|
#ifdef CONFIG_ARC_MMU_V3
|
|
#define TLB_DUP_ERR (TLB_LKUP_ERR | 0x00000001)
|
|
#else
|
|
#define TLB_DUP_ERR (TLB_LKUP_ERR | 0x40000000)
|
|
#endif
|
|
|
|
/*
|
|
* TLB Commands
|
|
*/
|
|
#define TLBWrite 0x1
|
|
#define TLBRead 0x2
|
|
#define TLBGetIndex 0x3
|
|
#define TLBProbe 0x4
|
|
#define TLBWriteNI 0x5 /* write JTLB without inv uTLBs */
|
|
#define TLBIVUTLB 0x6 /* explicitly inv uTLBs */
|
|
|
|
#ifdef CONFIG_ARC_MMU_V4
|
|
#define TLBInsertEntry 0x7
|
|
#define TLBDeleteEntry 0x8
|
|
#endif
|
|
|
|
/* Masks for actual TLB "PD"s */
|
|
#define PTE_BITS_IN_PD0 (_PAGE_GLOBAL | _PAGE_PRESENT | _PAGE_HW_SZ)
|
|
#define PTE_BITS_RWX (_PAGE_EXECUTE | _PAGE_WRITE | _PAGE_READ)
|
|
|
|
#define PTE_BITS_NON_RWX_IN_PD1 (PAGE_MASK_PHYS | _PAGE_CACHEABLE)
|
|
|
|
#ifndef __ASSEMBLY__
|
|
|
|
struct mm_struct;
|
|
extern int pae40_exist_but_not_enab(void);
|
|
|
|
static inline int is_pae40_enabled(void)
|
|
{
|
|
return IS_ENABLED(CONFIG_ARC_HAS_PAE40);
|
|
}
|
|
|
|
static inline void mmu_setup_asid(struct mm_struct *mm, unsigned long asid)
|
|
{
|
|
write_aux_reg(ARC_REG_PID, asid | MMU_ENABLE);
|
|
}
|
|
|
|
static inline void mmu_setup_pgd(struct mm_struct *mm, void *pgd)
|
|
{
|
|
/* PGD cached in MMU reg to avoid 3 mem lookups: task->mm->pgd */
|
|
#ifdef CONFIG_ISA_ARCV2
|
|
write_aux_reg(ARC_REG_SCRATCH_DATA0, (unsigned int)pgd);
|
|
#endif
|
|
}
|
|
|
|
#else
|
|
|
|
.macro ARC_MMU_REENABLE reg
|
|
lr \reg, [ARC_REG_PID]
|
|
or \reg, \reg, MMU_ENABLE
|
|
sr \reg, [ARC_REG_PID]
|
|
.endm
|
|
|
|
#endif /* !__ASSEMBLY__ */
|
|
|
|
#endif
|