Golang 内存分配优化 – 萌叔

1. 前言

imroc/reqio.Copyioutil.ReadAll

2. 起因

imroc/req

有如下调用关系
Resq.Bytes() -> Resq.ToBytes() -> ioutil.ReadAll(r io.Reader) -> Buff.ReadFrom(r io.Reader)

使用pprof收集内存累计分配情况可以发现大量的内存分配由Buffer.grow(n int) 触发

3. 原因分析

ReadFrom的源码如下

// MinRead is the minimum slice size passed to a Read call by
// Buffer.ReadFrom. As long as the Buffer has at least MinRead bytes beyond
// what is required to hold the contents of r, ReadFrom will not grow the
// underlying buffer.
const MinRead = 512
// ReadFrom reads data from r until EOF and appends it to the buffer, growing
// the buffer as needed. The return value n is the number of bytes read. Any
// error except io.EOF encountered during the read is also returned. If the
// buffer becomes too large, ReadFrom will panic with ErrTooLarge.
func (b *Buffer) ReadFrom(r io.Reader) (n int64, err error) {
    b.lastRead = opInvalid
    for {
        i := b.grow(MinRead)
        m, e := r.Read(b.buf[i:cap(b.buf)])
        if m < 0 {
            panic(errNegativeRead)
        }

        b.buf = b.buf[:i+m]
        n += int64(m)
        if e == io.EOF {
            return n, nil // e is EOF, so return nil explicitly
        }
        if e != nil {
            return n, e
        }
    }
}

MinReadBuffer.grow()

// grow grows the buffer to guarantee space for n more bytes.
// It returns the index where bytes should be written.
// If the buffer can't grow it will panic with ErrTooLarge.
func (b *Buffer) grow(n int) int {
    m := b.Len()
    // If buffer is empty, reset to recover space.
    if m == 0 && b.off != 0 {
        b.Reset()
    }
    // Try to grow by means of a reslice.
    if i, ok := b.tryGrowByReslice(n); ok {
        return i
    }
    // Check if we can make use of bootstrap array.
    if b.buf == nil && n <= len(b.bootstrap) {
        b.buf = b.bootstrap[:n]
        return 0
    }
    c := cap(b.buf)
    if n <= c/2-m {
        // We can slide things down instead of allocating a new
        // slice. We only need m+n <= c to slide, but
        // we instead let capacity get twice as large so we
        // don't spend all our time copying.
        copy(b.buf, b.buf[b.off:])
    } else if c > maxInt-c-n {
        panic(ErrTooLarge)
    } else {
        // Not enough space anywhere, we need to allocate.
        buf := makeSlice(2*c + n) // **注意这里**
        copy(buf, b.buf[b.off:])  
        b.buf = buf
    }
    // Restore b.off and len(b.buf).
    b.off = 0
    b.buf = b.buf[:m+n]
    return m
}

makeSlice(n int)

4.优化方案

makeSlice(n int)

为了验证方案，这里做出对比试验

1)未优化代码

2)优化代码

执行10,000次的操作的结果对比

指标	未优化方案	优化方案
mem.TotalAlloc(为堆对象总计分配的字节数)	186,170,656	42,164,192
mem.Mallocs(为创建堆对象总计的内存申请次数)	732,161	681,532
mem.Frees(为销毁堆对象总计的内存释放次数)	717,737	661,557

由上面图表知，临时对象的申请和释放次数都有明显下降，优化方案是有效的(事实上随着执行次数的增加，指标的差距还会进一步拉大)。

其它类似场景的优化

在某些情况下，你可能会用到io.Copy

io.Copy(dst Writer, src Reader)

io.Copyio.CopyBufferio.Copy

io.CopyBuffer(dst Writer, src Reader, buf []byte)

sync.Pool

请我喝瓶饮料