Centos7中systemd-journald占用内存过高问题

systemd-journald进程占用内存过高

同事反馈生产环境DB proxy代理层服务器间隔性地出现内存使用率过高告警，每次都通过重启部分进程处理，但过不了多久，问题会再次复现。

1.查看消耗memory的top进程

首先，在问题出现期间通过top命令按M可查看占用内存较高的top进程数，发现下面的systemd-journald进程消耗内存非常高，14G的机型单独该进程就占用了7.6G内存：

[root@dbhost ~]# top
top - 10:12:13 up 170 days, 22:27,  2 users,  load average: 2.22, 2.87, 3.41
Tasks: 423 total,   1 running, 390 sleeping,   1 stopped,  31 zombie
%Cpu(s): 11.3 us,  9.0 sy,  0.0 ni, 75.3 id,  0.2 wa,  0.0 hi,  3.7 si,  0.4 st
KiB Mem : 14027920 total,   547544 free, 11278808 used,  2201568 buff/cache
KiB Swap:        0 total,        0 free,        0 used.  2307820 avail Mem


  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                      
 8466 root      20   0 7957192   7.6g   5624 S   0.6 56.5 406:22.26 systemd-journal    <<<<< 占用7.6G                                                                                                                    
 1380 tdsql     20   0 3811756 645664   3328 S  44.9  4.6  32506:23 mysql-proxy                                                                                                                                  
30062 user_00   20   0  443428 248500    800 S   0.3  1.8 423:53.62 l5_agent                                                                                                                                     
 2438 tdsql     20   0 1042784  92972   3528 S   0.3  0.7   3330:18 mysql-proxy                                                                                                                                  
85161 tdsql     20   0  985404  86112   2272 S   0.6  0.6   3911:48 mysql-proxy                                                                                                                                  
54000 tdsql     20   0  862524  82176   2576 S   9.6  0.6  15644:05 mysql-proxy


[root@dbhost ~]# ps -ef | grep systemd-journald | grep -v grep
root      8466     1  0 Feb18 ?        06:46:28 usr/lib/systemd/systemd-journald
[root@dbhost ~]#

2.查看进程说明

通过man我们可以直接查看CentOS想法服务进程的介绍，systemd-journald的man信息大概就是说该服务进程用于收集并存储系统的登录信息。

[root@dbhost ~]# man systemd-journald
Description
systemd-journald is a system service that collects and stores logging data. 
It creates and maintains structured, indexed journals based on logging information that is received from a variety of sources:


Kernel log messages, via kmsg


Simple system log messages, via the libc syslog(3) call


Structured system log messages via the native Journal API, see sd_journal_print(3)


Standard output and standard error of service units. For further details see below.


Audit records, originating from the kernel audit subsystem

1.查看进程配置

既然是服务进程，我们不妨可以先看看改进程的配置文件，看看能否找到突破口，如下：

[root@dbhost ~]# cat etc/systemd/journald.conf
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See journald.conf(5) for details.


[Journal]
#Storage=auto    <<<该配置未开启
#Compress=yes
#Seal=yes
#SplitMode=uid
#SyncIntervalSec=5m
#RateLimitInterval=30s
#RateLimitBurst=1000
#SystemMaxUse=
#SystemKeepFree=
#SystemMaxFileSize=
RuntimeMaxUse=256M   <<<并不起作用
#RuntimeKeepFree=
#RuntimeMaxFileSize=
#MaxRetentionSec=
#MaxFileSec=1month
#ForwardToSyslog=yes
#ForwardToKMsg=no
#ForwardToConsole=no
#ForwardToWall=yes
#TTYPath=/dev/console
#MaxLevelStore=debug
#MaxLevelSyslog=debug
#MaxLevelKMsg=notice
#MaxLevelConsole=info
#MaxLevelWall=emerg
#LineMax=48K
[root@dbhost ~]#

2.查看配置说明

上面的配置信息中，可以看到该进程收集到的信息存储参数并没有配置，使用的是默认值auto，而默认值会/run/log/journal文件不存在的时候，会将信息保存在内存当中，。

[root@dbhost ~]# man 5 journald.conf
OPTIONS
       All options are configured in the "[Journal]" section:


       Storage=
           Controls where to store journal data. One of "volatile", "persistent", "auto" and "none". If "volatile", journal log data will be stored only in memory, i.e. below the run/log/journal
           hierarchy (which is created if needed). If "persistent", data will be stored preferably on disk, i.e. below the var/log/journal hierarchy (which is created if needed), with a fallback to
           run/log/journal (which is created if needed), during early boot and if the disk is not writable.  "auto" is similar to "persistent" but the directory var/log/journal is not created if needed,
           so that its existence controls where log data goes.  "none" turns off all storage, all log data received will be dropped. Forwarding to other targets, such as the console, the kernel log
           buffer, or a syslog socket will still work however. Defaults to "auto".

查看journal文件不存在

[root@dbhost ~]# ls -l var/log/journal
ls: cannot access var/log/journal: No such file or directory
[root@dbhost ~]#

四、解决方法及总结

把配置项Storage参数值改为none然后重启进程即可

1）把配置项Storage参数值改为none
[root@dbhost ~]# grep -i storage /etc/systemd/journald.conf
Storage=none
[root@dbhost ~]# 


2）重启进程生效
systemctl restart systemd-journald

总结

由于db代理层的服务器用户登录登出频繁，此时如果systemd-journald进程使用默认值，会将大量的用户登录信息存储到主机内存中，从而消耗大量的主机内存，如果日常运维管理对相关信息没要求，建议可以在CentOS 7版本中systemd-journal在主机初始化中优化掉。

Centos7中systemd-journald占用内存过高问题

原文：https://www.cnblogs.com/JerryTomcat/p/14742086.html