Python - uwsgi 응용 프로그램이 k8s 환경에서 OOM 발생하는 문제
직장 동료가 재미있는 오류를 하나 해결했습니다. ^^ uwsgi로 유독 k8s에 호스팅하는 경우에만 다음과 같은 예외가 발생하는 것인데요,
# cat uwsgi.ini
[uwsgi]
limit-as = 256 # limit the project to 256 MB
# uwsgi --http :8001 --chdir /app/my_tennis_club --ini ./uwsgi.ini
[uWSGI] getting INI configuration from ./uwsgi.ini
*** Starting uWSGI 2.0.24 (64bit) on [Thu Mar 21 05:37:03 2024] ***
compiled with version: 8.3.0 on 14 March 2024 07:42:21
os: Linux-5.15.150-1-MANJARO #1 SMP PREEMPT Fri Mar 1 16:56:12 UTC 2024
nodename: python-demo-deployment-f699c6585-n44h8
machine: x86_64
clock source: unix
detected number of CPU cores: 16
current working directory: /app
detected binary path: /usr/local/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
chdir() to /app/my_tennis_club
*** WARNING: you are running uWSGI without its master process manager ***
limiting address space of processes...
your process address space limit is 268435456 bytes (256 MB)
your memory page size is 4096 bytes
detected max file descriptor number: 1073741816
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uWSGI http bound on :8001 fd 4
spawned uWSGI http 1 (pid: 18)
uwsgi socket 0 bound to TCP address 127.0.0.1:39361 (port auto-assigned) fd 3
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
Python version: 3.11.4 (main, Jun 13 2023, 15:34:37) [GCC 8.3.0]
malloc(): Cannot allocate memory [core/utils.c line 1801]
!!! tried memory allocation of 8589934528 bytes !!!
*** backtrace of 18 ***
uwsgi(uwsgi_backtrace+0x2a) [0x55dde7709c5a]
uwsgi(uwsgi_malloc+0x5d) [0x55dde76c1eed]
uwsgi(uwsgi_corerouter_loop+0x36) [0x55dde772d556]
uwsgi(gateway_respawn+0x20a) [0x55dde770571a]
uwsgi(register_gateway+0x1c2) [0x55dde7705a12]
uwsgi(uwsgi_corerouter_init+0x9e) [0x55dde772ccbe]
uwsgi(http_init+0x33) [0x55dde772f933]
uwsgi(uwsgi_start+0x527) [0x55dde770b1a7]
uwsgi(uwsgi_setup+0x23a7) [0x55dde770e5c7]
uwsgi(main+0x9) [0x55dde76be7e9]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xeb) [0x7f3b321da09b]
uwsgi(_start+0x2a) [0x55dde76be81a]
*** end of backtrace ***
*** Python threads support is disabled. You can enable it with --enable-threads ***
Python main interpreter initialized at 0x7f3b328ff8f8
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 72904 bytes (71 KB) for 1 cores
*** Operational MODE: single process ***
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI worker 1 (and the only) (pid: 17, cores: 1)
원인을 다음의 글에서 찾아냈습니다.
uWSGI OOMKilled on Kubernetes #9562
; https://github.com/DefectDojo/django-DefectDojo/issues/9562
정리해 보면, uwsgi 실행 시 --max-fd 옵션을 이용해 file descriptor의 최댓값을 제한하는 것으로 해결했다는 것입니다.
사실, 저건 k8s의 문제만은 아닙니다. 다음의 글에 이와 관련한 문제가 또 나오는데요,
High memory usage if fs.nr_open is very high and no ulimit set on Linux systems #2299
; https://github.com/unbit/uwsgi/issues/2299
즉, uwsgi가 시스템에 설정된 file descriptor의 최댓값을 그대로 적용해 내부적으로 메모리를 할당하는 문제가 있는 것입니다. 실제로 위에서 예를 든 k8s의 문제가 되었던 uwsgi 로그 출력 내용에,
...[생략]...
detected max file descriptor number: 1073741816
...[생략]...
라는 문구가 나오는데요, 결국 시스템에 저 값이 설정돼 있기 때문에 uwsgi는 그에 따라 반응한 것입니다. 따라서, 이 문제는 단순히 uwsgi의 --max-fd 옵션으로 해결할 것이 아니라, 어차피 해당 옵션으로 제한할 수 있다면 애당초 uwsgi 스스로 내부에서 file descriptor를 위한 기본값을 1048576 정도로 낮게 설정해 두는 것이 더 나을 수 있었을 것입니다.
그나저나, Windows의 Ubutun 20.04 WSL 환경에서는 이 값이,
$ sysctl fs.nr_open
fs.nr_open = 1048576
1048576로 나오는데요, uwsgi의 출력 화면에서는 여전히 "detected max file descriptor number: 1024"로 잡힙니다.
별도로 설치한 Ubuntu 18.04의 경우에도 fs.nr_open은 1048576이었는데, 문제가 되었던 1073741816으로 대폭 상향시켜도,
$ sudo sysctl -w fs.nr_open=1073741816
$ django-admin startproject testdj
$ uwsgi --http :8000 --chdir ./testdj/ --wsgi-file ./testdj/wsgi.py
...[생략]...
detected max file descriptor number: 1048576
...[생략]...
여전히 uwsgi는 1048576 값을 출력합니다. 뭔가, 이쪽도 알 수 없는 설정 절차가 있는 듯합니다. ^^;
참고로, Ubuntu 18.04 테스트 머신에서 "sysctl -w fs.nr_open=1073741816" 설정 후 재부팅하고 나면 다시 1048576으로 돌아옵니다. 관련 sysctl.conf 파일의 내용을 보면 이렇고,
$ cat /etc/sysctl.conf
#
# /etc/sysctl.conf - Configuration file for setting system variables
# See /etc/sysctl.d/ for additional system variables.
# See sysctl.conf (5) for information.
#
#kernel.domainname = example.com
# Uncomment the following to stop low-level messages on console
#kernel.printk = 3 4 1 3
##############################################################3
# Functions previously found in netbase
#
# Uncomment the next two lines to enable Spoof protection (reverse-path filter)
# Turn on Source Address Verification in all interfaces to
# prevent some spoofing attacks
#net.ipv4.conf.default.rp_filter=1
#net.ipv4.conf.all.rp_filter=1
# Uncomment the next line to enable TCP/IP SYN cookies
# See http://lwn.net/Articles/277146/
# Note: This may impact IPv6 TCP sessions too
#net.ipv4.tcp_syncookies=1
# Uncomment the next line to enable packet forwarding for IPv4
#net.ipv4.ip_forward=1
# Uncomment the next line to enable packet forwarding for IPv6
# Enabling this option disables Stateless Address Autoconfiguration
# based on Router Advertisements for this host
#net.ipv6.conf.all.forwarding=1
###################################################################
# Additional settings - these settings can improve the network
# security of the host and prevent against some network attacks
# including spoofing attacks and man in the middle attacks through
# redirection. Some network environments, however, require that these
# settings are disabled so review and enable them as needed.
#
# Do not accept ICMP redirects (prevent MITM attacks)
#net.ipv4.conf.all.accept_redirects = 0
#net.ipv6.conf.all.accept_redirects = 0
# _or_
# Accept ICMP redirects only for gateways listed in our default
# gateway list (enabled by default)
# net.ipv4.conf.all.secure_redirects = 1
#
# Do not send ICMP redirects (we are not a router)
#net.ipv4.conf.all.send_redirects = 0
#
# Do not accept IP source route packets (we are not a router)
#net.ipv4.conf.all.accept_source_route = 0
#net.ipv6.conf.all.accept_source_route = 0
#
# Log Martian Packets
#net.ipv4.conf.all.log_martians = 1
#
###################################################################
# Magic system request Key
# 0=disable, 1=enable all
# Debian kernels have this set to 0 (disable the key)
# See https://www.kernel.org/doc/Documentation/sysrq.txt
# for what other values do
#kernel.sysrq=1
###################################################################
# Protected links
#
# Protects against creating or following links under certain conditions
# Debian kernels have both set to 1 (restricted)
# See https://www.kernel.org/doc/Documentation/sysctl/fs.txt
#fs.protected_hardlinks=0
#fs.protected_symlinks=0
/etc/sysctl.d 디렉터리에는 이런 파일들이 있는데,
$ ls /etc/sysctl.d
10-console-messages.conf 10-kernel-hardening.conf 10-magic-sysrq.conf 10-ptrace.conf 99-sysctl.conf
10-ipv6-privacy.conf 10-link-restrictions.conf 10-network-security.conf 10-zeropage.conf README
그중에 99-sysctl.conf 파일이 /etc/sysctl.conf와 링크가 된 파일입니다. 그래서, 일단 /etc/sysctl.conf 파일의 마지막에,
$ cat /etc/sysctl.conf
...[생략]...
fs.nr_open=1073741816
값을 직접 기록하고 재부팅을 했더니 저 값이 유지됩니다. (찾아보니까 "sysctl -p" 옵션으로 재부팅 없이 반영할 수 있다고 합니다.)
그렇다 해도 여전히 uwsgi는 1048576으로 설정이 잡히는데요, 휴~~~ 일부러 1073741816 값으로 인한 오동작을 재현하려고 해도 쉽지가 않군요. ^^;
마지막으로, fs.nr_open을 확인하는 명령어에서 단순히 fs만 주면 모든 값을 볼 수 있습니다.
$ sudo sysctl fs
[sudo] password for kevin:
fs.aio-max-nr = 65536
fs.aio-nr = 0
fs.binfmt_misc.status = enabled
fs.dentry-state = 89220 72200 45 0 47968 0
fs.dir-notify-enable = 1
fs.epoll.max_user_watches = 822005
fs.file-max = 400247
fs.file-nr = 2432 0 400247
fs.inode-nr = 42041 905
fs.inode-state = 42041 905 0 0 0 0 0
fs.inotify.max_queued_events = 16384
fs.inotify.max_user_instances = 128
fs.inotify.max_user_watches = 8192
fs.lease-break-time = 45
fs.leases-enable = 1
fs.mount-max = 100000
fs.mqueue.msg_default = 10
fs.mqueue.msg_max = 10
fs.mqueue.msgsize_default = 8192
fs.mqueue.msgsize_max = 8192
fs.mqueue.queues_max = 256
fs.nr_open = 1048576
fs.overflowgid = 65534
fs.overflowuid = 65534
fs.pipe-max-size = 1048576
fs.pipe-user-pages-hard = 0
fs.pipe-user-pages-soft = 16384
fs.protected_fifos = 0
fs.protected_hardlinks = 1
fs.protected_regular = 0
fs.protected_symlinks = 1
fs.quota.allocated_dquots = 0
fs.quota.cache_hits = 0
fs.quota.drops = 0
fs.quota.free_dquots = 0
fs.quota.lookups = 0
fs.quota.reads = 0
fs.quota.syncs = 0
fs.quota.warnings = 1
fs.quota.writes = 0
fs.suid_dumpable = 0
fs.xfs.error_level = 3
fs.xfs.filestream_centisecs = 3000
fs.xfs.inherit_noatime = 1
fs.xfs.inherit_nodefrag = 1
fs.xfs.inherit_nodump = 1
fs.xfs.inherit_nosymlinks = 0
fs.xfs.inherit_sync = 1
fs.xfs.irix_sgid_inherit = 0
fs.xfs.irix_symlink_mode = 0
fs.xfs.panic_mask = 0
fs.xfs.rotorstep = 1
fs.xfs.speculative_cow_prealloc_lifetime = 1800
fs.xfs.speculative_prealloc_lifetime = 300
fs.xfs.stats_clear = 0
fs.xfs.xfssyncd_centisecs = 3000
[이 글에 대해서 여러분들과 의견을 공유하고 싶습니다. 틀리거나 미흡한 부분 또는 의문 사항이 있으시면 언제든 댓글 남겨주십시오.]