aws beanstalk 实例日志和环境状态日志的轮换和流式传输配置
参考资料
-
https://docs.aws.amazon.com/zh_cn/elasticbeanstalk/latest/dg/AWSHowTo.cloudwatchlogs.html
-
查看您的 Elastic Beanstalk 环境中的 Amazon EC2 实例的日志
-
https://docs.aws.amazon.com/zh_cn/elasticbeanstalk/latest/dg/environments-cfg-logging.html
eb相关日志的说明
日志源
beanstalk环境中的相关日志主要包括以下几个
-
实例日志
-
Web 服务器
-
应用程序服务器
-
beanstalk平台脚本和
-
cloudformation创建日志
-
-
环境状态日志
日志分类
按照日志的类型又可以分为
- 结尾日志,beanstalk的操作日志和webserver日志,将条目保存为文本文件
- 捆绑日志(windows平台不支持),yum 和 cron日志,cloudformation日志,打包为zip包
日志上传方式
日志的上传也有两种方式
-
将轮换的实例日志上传到环境的s3存储桶
-
将实例日志流式传输到CloudWatch Logs,相关配置项如下(包括实例日志和环境状态日志)
aws:elasticbeanstalk:cloudwatch:logs: DeleteOnTerminate: 'false' RetentionInDays: '7' StreamLogs: 'false' aws:elasticbeanstalk:cloudwatch:logs:health: DeleteOnTerminate: 'false' HealthStreamingEnabled: 'false' RetentionInDays: '7'
日志配置项
日志配置主要涉及到的命名空间如下
- aws:elasticbeanstalk:hostmanager - 配置为将轮换日志上传到 Amazon S3
- aws:elasticbeanstalk:cloudwatch:logs - 配置为将实例日志流式传输到 CloudWatch
- aws:elasticbeanstalk:cloudwatch:logs:health - 配置为将环境运行状况流式传输到 CloudWatch
实例日志的配置
可以直接在命令行快速开启
eb logs --log-group xxxxx
eb logs --cloudwatch-logs enable --cloudwatch-log-source instance
eb logs --cloudwatch-logs enable --cloudwatch-log-source environment-health
eb logs --stream
流日志
在应用根目录增加以下配置,aws:elasticbeanstalk:cloudwatch:logs
$ cat .ebextensions/cloudwatch.config
option_settings:
aws:elasticbeanstalk:cloudwatch:logs:
DeleteOnTerminate: 'false'
RetentionInDays: '1'
StreamLogs: 'true'
配置完成后控制台可以看到日志组名称
默认收集的日志如下
s3日志
在应用根目录增加以下配置,aws:elasticbeanstalk:hostmanager
$ cat .ebextensions/hostmanagerlog.config
option_settings:
aws:elasticbeanstalk:hostmanager:
LogPublicationControl: true
上传的日志清单为
https://docs.aws.amazon.com/zh_cn/elasticbeanstalk/latest/dg/using-features.logging.html
查看应用对应的s3桶,eb实例每小时上传一次轮换日志
-
如果应用程序不是在环境平台默认配置的位置生成日志,可以使用配置文件 (
.ebextensions
) 扩展默认配置。 -
也可以将应用程序的日志文件添加到结尾日志、捆绑日志或日志轮换中
s3日志的存储位置为,名为elasticbeanstalk-region-account-id
的存储桶,路径为resources/environments/logs/
logtype/
environment-id/
instance-id``
日志名称规范为
- 结尾日志 –
s3://elasticbeanstalk-us-west-2-123456789012/resources/environments/logs/tail/e-mpcwnwheky/i-0a1fd158
- 捆绑日志 –
s3://elasticbeanstalk-us-west-2-123456789012/resources/environments/logs/bundle/e-mpcwnwheky/i-0a1fd158
- 轮换日志 –
s3://elasticbeanstalk-us-west-2-123456789012/resources/environments/logs/publish/e-mpcwnwheky/i-0a1fd158
结尾日志和捆绑日志创建15分钟后删除(eb使用用户权限删除s3)
对于轮换日志,具体配置在/etc/logrotate.elasticbeanstalk.hourly/
中
$ ll /etc/logrotate.elasticbeanstalk.hourly
total 28
-rw-r--r-- 1 root root 157 Apr 1 11:11 logrotate.elasticbeanstalk.eb-engine.conf
-rw-r--r-- 1 root root 156 Apr 1 11:11 logrotate.elasticbeanstalk.eb-hooks.conf
-rw-r--r-- 1 root root 171 Apr 1 11:45 logrotate.elasticbeanstalk.healthd.conf
-rw-r--r-- 1 root root 157 Apr 1 11:45 logrotate.elasticbeanstalk.nginx.conf
-rw-r--r-- 1 root root 156 Apr 1 11:11 logrotate.elasticbeanstalk.web-stderr.conf
-rw-r--r-- 1 root root 156 Apr 1 11:11 logrotate.elasticbeanstalk.web-stdout.conf
-rw-r--r-- 1 root root 162 Apr 1 11:45 logrotate.elasticbeanstalk.xray.conf
查看eb-engine日志配置
$ cat logrotate.elasticbeanstalk.eb-engine.conf
/var/log/eb-engine.log {
su root root
size 10M
rotate 5
missingok
compress
notifempty
copytruncate
dateext
dateformat %s
olddir /var/log/rotated
}
这些配置文件由定时任务配置进行调用
$ ls /etc/cron.hourly/
0anacron cron.logrotate.elasticbeanstalk.eb-hooks.conf cron.logrotate.elasticbeanstalk.web-stderr.conf
cron.logcleanup.elasticbeanstalk.healthd-proxy.conf cron.logrotate.elasticbeanstalk.healthd.conf cron.logrotate.elasticbeanstalk.web-stdout.conf
cron.logrotate.elasticbeanstalk.eb-engine.conf cron.logrotate.elasticbeanstalk.nginx.conf cron.logrotate.elasticbeanstalk.xray.conf
查看定时任务配置,可以看到关键程序为logrotate
$ cat /etc/cron.hourly/cron.logrotate.elasticbeanstalk.eb-engine.conf
#!/bin/sh
test -x /usr/sbin/logrotate || exit 0
/usr/sbin/logrotate -f /etc/logrotate.elasticbeanstalk.hourly/logrotate.elasticbeanstalk.eb-engine.conf
可以对默认的日志位置进行扩展,具体的配置路径
- 结尾日志 –
/opt/elasticbeanstalk/tasks/taillogs.d/
- 捆绑日志 –
/opt/elasticbeanstalk/tasks/bundlelogs.d/
- 轮换日志 –
/opt/elasticbeanstalk/tasks/publishlogs.d/
扩展其实就是在.ebextension路径下在配置文件中设定日志配置的文件和内容
files:
"/opt/elasticbeanstalk/tasks/taillogs.d/cwl-system.conf":
content: |
/var/log/awslogs-agent-setup.log
/var/log/awslogs.log
/var/log/eb-cwlogs.log
mode : "000644"
"/opt/elasticbeanstalk/tasks/bundlelogs.d/cwl-system.conf":
content: |
/var/log/awslogs-agent-setup.log
/var/log/awslogs.log
/var/log/eb-cwlogs.log
mode : "000644"
"/opt/elasticbeanstalk/tasks/systemtaillogs.d/cwl-system.conf":
content: |
/var/log/awslogs-agent-setup.log
/var/log/awslogs.log
/var/log/eb-cwlogs.log
mode : "000644"
"/opt/elasticbeanstalk/tasks/publishlogs.d/cwl-system.conf":
content: |
/var/log/awslogs-agent-setup.log ## this isn't rotated
/var/log/awslogs.log*.gz
mode : "000644"
环境状态日志的配置
在应用根目录增加以下配置
$ cat .ebextensions/cloudwatch.config
option_settings:
aws:elasticbeanstalk:healthreporting:system:
SystemType: enhanced
aws:elasticbeanstalk:cloudwatch:logs:health:
DeleteOnTerminate: 'false'
HealthStreamingEnabled: 'true'
RetentionInDays: '1'
配置完成后控制台可以看到日志组名称
日志配置相关的仓库
- https://github.com/awsdocs/elastic-beanstalk-samples/blob/master/configuration-files/aws-provided/instance-configuration/awslogs-change-frequency.config
- https://github.com/awsdocs/elastic-beanstalk-samples/blob/master/configuration-files/aws-provided/instance-configuration/logs-streamtocloudwatch-linux-alami.config
- https://github.com/awsdocs/elastic-beanstalk-samples/blob/master/configuration-files/aws-provided/instance-configuration/logs-streamtocloudwatch-linux.config
- https://github.com/awsdocs/elastic-beanstalk-samples/blob/master/configuration-files/aws-provided/instance-configuration/logs-uploadonterminate-linux.config