未加星标

nodejs CorkedRequest导致内存泄露

字体大小 | |
[前端(javascript) 所属分类 前端(javascript) | 发布者 店小二03 | 时间 2016 | 作者 红领巾 ] 0人收藏点击收藏
0x00 状态

有一个websocket长连接服务原先是php的workerman框架跑的,但是近期出现了一些莫名其妙的bug,用Nodejs重写后接替了原先的全部流量。

最初还没什么问题。

跑了一段时间后发现内存泄露问题比较严重,服务器有8G内存,用cluster方式跑了8个进程,几乎24小时内就会吃满8g内存,导致进程被系统kill。

0x01 堆内存分析

遇到内存泄露,第一时间想到的就是把内存dump下来看看,于是发布了一个能够实时dump内存的版本,拿到heapdump文件后发现文件只有60M不到,而当时的进程内存占用达到了1.6G,于是能够知道大量的内存开销都不是堆内存,而是堆外内存占用。

将dump文件交给webstorm分析,得到的结果非常诡异


nodejs CorkedRequest导致内存泄露
nodejs CorkedRequest导致内存泄露

有大量的Buffer对象是已经猜到的,但是其中大量的Buffer有着成千上万的引用深度,点开Webstorm直接卡死,分析只有几百的引用深度的对象,发现引用树几乎一直在WriteReq.next()方法之间自己调用自己,几乎没我业务代码什么事。

Google关键字”nodejs writereq memory leak”。通常来说,Google也解决不了的,只有两种情况,一是低级bug,二是真的前无古人的bug,而大部分都是低级bug。

但是过了几天也依旧没想到其中原因。

0x02 线索

又过了几天,发现一个规律,在高峰期带宽不太足的时候内存增长非常快。

但是按正常情况下,带宽不足时会在内存中缓存发送的数据是很正常的,但是如果有将近8G的数据堆在内存里等待发送的话,客户端应该会有明显的感觉。但是事实是客户端几乎没有太多的延迟感。

0x03 源码

最终没有办法了,决定阅读nodejs的源码。

WriteReq的定义在_stream_writeable.js中,是一个nodejs内部私有模块 https://github.com/nodejs/node/blob/master/lib/_stream_writable.js

function WriteReq(chunk, encoding, cb) { this.chunk = chunk; this.encoding = encoding; this.callback = cb; this.next = null; } Writable.prototype.write = function(chunk, encoding, cb) { var state = this._writableState; var ret = false; if (typeof encoding === 'function') { cb = encoding; encoding = null; } if (chunkinstanceof Buffer) encoding = 'buffer'; else if (!encoding) encoding = state.defaultEncoding; if (typeof cb !== 'function') cb = nop; if (state.ended) writeAfterEnd(this, cb); else if (validChunk(this, state, chunk, cb)) { state.pendingcb++; ret = writeOrBuffer(this, state, chunk, encoding, cb); } return ret; }; function writeOrBuffer(stream, state, chunk, encoding, cb) { chunk = decodeChunk(state, chunk, encoding); if (chunkinstanceof Buffer) encoding = 'buffer'; var len = state.objectMode ? 1 : chunk.length; state.length += len; var ret = state.length < state.highWaterMark; // we must ensure that previous needDrain will not be reset to false. if (!ret) state.needDrain = true; if (state.writing || state.corked) { var last = state.lastBufferedRequest; state.lastBufferedRequest = new WriteReq(chunk, encoding, cb); if (last) { last.next = state.lastBufferedRequest; } else { state.bufferedRequest = state.lastBufferedRequest; } state.bufferedRequestCount += 1; } else { doWrite(stream, state, false, len, chunk, encoding, cb); } return ret; }

WriteableStream的write方法在调用时如果发现上次的write还未结束,会构建WriteReq对象链,WriteReq.next指向本次写入新生成的WriteReq。于是就形成了


nodejs CorkedRequest导致内存泄露

到这里为止还算正常,那么接下来应该看看如果WriteReq被写了会怎么样

// It seems a linked list but it is not // there will be only 2 of these for each stream function CorkedRequest(state) { this.next = null; this.entry = null; this.finish = (err) => { var entry = this.entry; this.entry = null; while (entry) { var cb = entry.callback; state.pendingcb--; cb(err); entry = entry.next; } if (state.corkedRequestsFree) { state.corkedRequestsFree.next = this; } else { state.corkedRequestsFree = this; } }; }

当一个上WriteReq对象在写入完成后会调用socket.state.onwrite,socket.write是一个WritableState对象,他的onwrite方法是

function onwrite(stream, er) { var state = stream._writableState; var sync = state.sync; var cb = state.writecb; onwriteStateUpdate(state); if (er) onwriteError(stream, state, sync, er, cb); else { // Check if we're actually ready to finish, but don't emit yet var finished = needFinish(state); if (!finished && !state.corked && !state.bufferProcessing && state.bufferedRequest) { clearBuffer(stream, state); } if (sync) { process.nextTick(afterWrite, stream, state, finished, cb); } else { afterWrite(stream, state, finished, cb); } } }

在这个方法里可以看到,只有stream缓冲流被清空,没有任何等待发送的数据,this.bufferedRequest == null时,才会调用clearBuffer方法

而clearBuffer干了什么呢

function clearBuffer(stream, state) { state.bufferProcessing = true; var entry = state.bufferedRequest; if (stream._writev && entry && entry.next) { // Fast case, write everything using _writev() var l = state.bufferedRequestCount; var buffer = new Array(l); var holder = state.corkedRequestsFree; holder.entry = entry; var count = 0; while (entry) { buffer[count] = entry; entry = entry.next; count += 1; } doWrite(stream, state, true, state.length, buffer, '', holder.finish); // doWrite is almost always async, defer these to save a bit of time // as the hot path ends with doWrite state.pendingcb++; state.lastBufferedRequest = null; if (holder.next) { state.corkedRequestsFree = holder.next; holder.next = null; } else { state.corkedRequestsFree = new CorkedRequest(state); } } else { // Slow case, write chunks one-by-one while (entry) { var chunk = entry.chunk; var encoding = entry.encoding; var cb = entry.callback; var len = state.objectMode ? 1 : chunk.length; doWrite(stream, state, false, len, chunk, encoding, cb); entry = entry.next; // if we didn't call the onwrite immediately, then // it means that we need to wait until it does. // also, that means that the chunk and cb are currently // being processed, so move the buffer counter past them. if (state.writing) { break; } } if (entry === null) state.lastBufferedRequest = null; } state.bufferedRequestCount = 0; state.bufferedRequest = entry; state.bufferProcessing = false; }

clearBuffer函数里清理掉了整条WriteReq链,并且调用了socket.write()的回调

那么就会有一个问题,如果程序数据生成的速度大于网络带宽时,由于流永远不会退出缓冲模式,所以clearBuffer方法也就永远不会被调用到,也就永远不会有人清理WriteReq链,于是造成了WriteReq链上持有的buffer对象会一直堆积在内存里无法释放。

Buffer对象存储数据所用的内存是不在V8的堆内存内的,也就吻合了1.6G的内存占用dump只有60M的情况。

0x04 解决方案

解决方案其实也很简单暴力……在node前面用nginx跑一层反代,不要让node把请求缓存起来就OK了

https://cnodejs.org/topic/57e90fb256898f231a526f7b#5806efe027a1d99178a98fa5

本文前端(javascript)相关术语:javascript是什么意思 javascript下载 javascript权威指南 javascript基础教程 javascript 正则表达式 javascript设计模式 javascript高级程序设计 精通javascript javascript教程

主题: 服务器数据中大其实
分页:12
转载请注明
本文标题:nodejs CorkedRequest导致内存泄露
本站链接:http://www.codesec.net/view/485527.html
分享请点击:


1.凡CodeSecTeam转载的文章,均出自其它媒体或其他官网介绍,目的在于传递更多的信息,并不代表本站赞同其观点和其真实性负责;
2.转载的文章仅代表原创作者观点,与本站无关。其原创性以及文中陈述文字和内容未经本站证实,本站对该文以及其中全部或者部分内容、文字的真实性、完整性、及时性,不作出任何保证或承若;
3.如本站转载稿涉及版权等问题,请作者及时联系本站,我们会及时处理。
登录后可拥有收藏文章、关注作者等权限...
技术大类 技术大类 | 前端(javascript) | 评论(0) | 阅读(31)