Linux grep技巧 提取log中的json数据

一. 前提

1.1 数据准备

545-1 2024/07/20 18:20:21 [ERROR] MPX001 eventController=aupay transactionId=A545 {"event":"ERROR","auid":"kddi_XXXasd1","category":{"id":"110","name":"APPLE 1","amout":10,"price":12}}
545-2 2024/07/20 18:20:22 [INFO] MPX001 eventController=alipay transactionId=Bljk {"event":"INFO","auid":"kddi_XXXasd2","category":{"id":"111","name":"APPLE 2","amout":11,"price":13}}
545-3 2024/07/20 18:20:23 [ERROR] MPX001 eventController=paypay transactionId=Ijkhjk {"event":"ERROR","auid":"kddi_XXXasd3","category":{"id":"112","name":"APPLE 3","amout":12,"price":14}}
545-4 2024/07/20 18:20:24 [INFO] MPX001 eventController=alipay transactionId=C9joj {"event":"INFO","auid":"kddi_XXXasd4","category":{"id":"113","name":"APPLE 4","amout":13,"price":15}}

1.2 需求

• 从log中提取出[ERROR]相关的日志
• 然后再从日志中进一步提取eventController，transactionId，JSON数据中的auid和name
• 服务器中没有安装jq库，无法通过此种方式来处理json数据。

1.3 分析

• ①先提取[ERROR]相关的日志，过滤掉INFO的日志
• ②再提取出eventController和transactionId字段
• ③然后可以先尝试提取出所有的json数据
  • 使用{.*}正则表达式
• ④然后可以进一步缩小范围，提取category相关的json数据
  • 使用"category":{.*}正则表达式
• ⑤然后可以进一步提取auid和name所对应的值
  • auid":"[^"]*"
  • name":"[^"]*"
• ⑥最后再通过sed命令将各字段转置到一行上
  • sed ':loop; N; $!b loop; ;s/\n\([tan]\)/ \1/g'

二. 数据提取

2.1 提取所有的json数据

• {.*}

grep -E "\[ERROR\]\sMPX001" ./result.log | \
grep -o -e "eventController=\S*" -e "transactionId=\S*" -e "{.*}" | \
sed ':loop; N; $!b loop; ;s/\n\([t{]\)/ \1/g'

⏹效果如下

fengyehong@ubuntu:~/jmw_work_space/20270720$ grep -E "\[ERROR\]\sMPX001" ./result.log | \
> grep -o -e "eventController=\S*" -e "transactionId=\S*" -e "{.*}" | \
> sed ':loop; N; $!b loop; ;s/\n\([t{]\)/ \1/g'eventController=aupay transactionId=A545 {"event":"ERROR","auid":"kddi_XXXasd1","category":{"id":"110","name":"APPLE 1","amout":10,"price":12}}
eventController=paypay transactionId=Ijkhjk {"event":"ERROR","auid":"kddi_XXXasd3","category":{"id":"112","name":"APPLE 3","amout":12,"price":14}}

2.2 提取子项目的全部json数据

• "category":{.*}

grep -E "\[ERROR\]\sMPX001" ./result.log | \
grep -o -e "eventController=\S*" -e "transactionId=\S*" -e '"category":{.*}' | \
sed ':loop; N; $!b loop; ;s/\n\([t"]\)/ \1/g'

⏹效果如下

fengyehong@ubuntu:~/jmw_work_space/20270720$ grep -E "\[ERROR\]\sMPX001" ./result.log | \
> grep -o -e "eventController=\S*" -e "transactionId=\S*" -e '"category":{.*}' | \
> sed ':loop; N; $!b loop; ;s/\n\([t"]\)/ \1/g'eventController=aupay transactionId=A545 "category":{"id":"110","name":"APPLE 1","amout":10,"price":12}}
eventController=paypay transactionId=Ijkhjk "category":{"id":"112","name":"APPLE 3","amout":12,"price":14}}

2.3 提取指定项目的json数据

• auid":"[^"]*"
• name":"[^"]*"
  • [ 和 ]：定义一个字符类。
  • ^：在字符类中表示否定，意味着字符类匹配所有不在括号内的字符。
  • "：字符类中唯一被否定的字符。
  • *：表示前面的模式（即 [^"]）可以出现零次或多次。

因此，[^"]* 表示匹配由任意数量的非双引号字符组成的字符串，包括零个字符的情况。

grep -E "\[ERROR\]\sMPX001" ./result.log | \
grep -o -e "eventController=\S*" -e "transactionId=\S*" -e 'auid":"[^"]*"' -e 'name":"[^"]*"' | \
sed ':loop; N; $!b loop; ;s/\n\([tan]\)/ \1/g'

⏹效果如下

fengyehong@ubuntu:~/jmw_work_space/20270720$ grep -E "\[ERROR\]\sMPX001" ./result.log | \
> grep -o -e "eventController=\S*" -e "transactionId=\S*" -e 'auid":"[^"]*"' -e 'name":"[^"]*"' | \
> sed ':loop; N; $!b loop; ;s/\n\([tan]\)/ \1/g'eventController=aupay transactionId=A545 auid":"kddi_XXXasd1" name":"APPLE 1"
eventController=paypay transactionId=Ijkhjk auid":"kddi_XXXasd3" name":"APPLE 3"