server.max-http-header-size设置不当引发的线上OOM案例分析

问题现象

后台服务日志，大量报出如下异常，关键字：java.lang.OutOfMemoryError: Java heap space，问题指向：o.a.c.h.Http11NioProtocol [DirectJDKLog.java:175] Failed to complete processing of a request

问题定位

线上OOM后，dump内存快照，通过MAT工具进行分析，如下图所示，有多个200MB的Http11OutputBuffer对象存在，这很不合理。

排查相关配置后发现，server.max-http-header-size被设置为200MB。

问题解决

将server.max-http-header-size调整为1MB即可。

问题分析

Tomcat在构建Http11OutputBuffer对象时会通过maxHttpHeaderSize来设置HTTP message header的大小，其默认为8KB，并通过ByteBuffer.allocate(headerBufferSize)向内存申请，也就是说每接收到一个HTTP请求，都会向内存申请200MB的空间，所以，当请求并发上来以后，必然会导致OOM。

源码

org.apache.coyote.http11.AbstractHttp11Protocol类，构建Processor对象

@Override
protected Processor createProcessor() {
    Http11Processor processor = new Http11Processor(this, adapter);
    return processor;
}

Http11Processor构建

public Http11Processor(AbstractHttp11Protocol<?> protocol, Adapter adapter) {
    super(adapter);
    this.protocol = protocol;
    httpParser = new HttpParser(protocol.getRelaxedPathChars(),
            protocol.getRelaxedQueryChars());
    inputBuffer = new Http11InputBuffer(request, protocol.getMaxHttpHeaderSize(),
            protocol.getRejectIllegalHeader(), httpParser);
    request.setInputBuffer(inputBuffer);
    // getMaxHttpHeaderSize默认为8KB，也可以通过server.max-http-header-size进行修改
    outputBuffer = new Http11OutputBuffer(response, protocol.getMaxHttpHeaderSize());
    response.setOutputBuffer(outputBuffer);
    // Create and add the identity filters.
    inputBuffer.addFilter(new IdentityInputFilter(protocol.getMaxSwallowSize()));
    outputBuffer.addFilter(new IdentityOutputFilter());
    // Create and add the chunked filters.
    inputBuffer.addFilter(new ChunkedInputFilter(protocol.getMaxTrailerSize(),
            protocol.getAllowedTrailerHeadersInternal(), protocol.getMaxExtensionSize(),
            protocol.getMaxSwallowSize()));
    outputBuffer.addFilter(new ChunkedOutputFilter());
    // Create and add the void filters.
    inputBuffer.addFilter(new VoidInputFilter());
    outputBuffer.addFilter(new VoidOutputFilter());
    // Create and add buffered input filter
    inputBuffer.addFilter(new BufferedInputFilter());
    // Create and add the gzip filters.
    //inputBuffer.addFilter(new GzipInputFilter());
    outputBuffer.addFilter(new GzipOutputFilter());
    pluggableFilterIndex = inputBuffer.getFilters().length;
}

protected Http11OutputBuffer(Response response, int headerBufferSize) {
    this.response = response;
    // 内存申请
    headerBuffer = ByteBuffer.allocate(headerBufferSize);
    filterLibrary = new OutputFilter[0];
    activeFilters = new OutputFilter[0];
    lastActiveFilter = -1;
    responseFinished = false;
    outputStreamOutputBuffer = new SocketOutputBuffer();
}