Netty 4源码解析：服务端启动

Netty 4源码解析：服务端启动

1.基础知识

1.1 Netty 4示例

因为Netty 5还处于测试版，所以选择了目前比较稳定的Netty 4作为学习对象。而且5.0的变化也不像4.0这么大，好多网上的例子都已经过时了。

<dependency>
            <groupId>io.netty</groupId>
            <artifactId>netty-all</artifactId>
            <version>4.0.25.Final</version>
        </dependency>

Netty 4服务端的典型用法如下面代码示例所示，核心组件就是EventLoopGroup、ServerBootstrap、Handler等。其中像EventLoopGroup、Channel等都是可以灵活调配的。这里以比较常用的“主从Reactor”+Nio非阻塞为例，分析代码的执行流程。如果没有接触过Netty的话，建议先简单了解一下Reactor模型等知识再学习源码，不然可能会一头雾水。

EventLoopGroup bossGroup = new NioEventLoopGroup();
    EventLoopGroup workerGroup = new NioEventLoopGroup();
    try {
        ServerBootstrap b = new ServerBootstrap()
                .group(bossGroup, workerGroup)
                .channel(NioServerSocketChannel.class)
                .localAddress(port)
                .childHandler(new ChannelInitializer<SocketChannel>() {
                    @Override
                    public void initChannel(SocketChannel ch) throws Exception {
                        ch.pipeline().addLast(new XXXHandler());
                    }
                });

        // Bind and start to accept incoming connections.
        ChannelFuture f = b.bind(port).sync();

        // Wait until the server socket is closed.
        f.channel().closeFuture().sync();
    } finally {
        bossGroup.shutdownGracefully();
        workerGroup.shutdownGracefully();
    }

1.2 NIO示例

不管用Netty还是其他网络框架，最终都绕不开JDK NIO提供的接口。那直接用NIO可以分为几步呢？

Selector.open()：创建当前平台的Selector。

ServerSocketChannel.open()：创建服务端的Channel。

bind()：绑定到某个端口上。

register()：注册Channel和关注的事件到Selector上。

select()：拿到已经就绪的事件。

下面就是一段NIO的示例代码，用单线程和一个Selector监控两个Channel的事件。

public static void main(String[] args) throws Exception {
        Selector selector = Selector.open();

        int[] ports = { 1234, 5678 };
        for (int port : ports) {
            ServerSocketChannel listenChannel = ServerSocketChannel.open();
            listenChannel.socket().bind(new InetSocketAddress("localhost", port));
            listenChannel.configureBlocking(false);
            listenChannel.register(selector, SelectionKey.OP_ACCEPT);
        }

        while (true) {
            if (selector.select(3000) == 0) {
                System.out.print(".");
                continue;
            }

            Iterator<SelectionKey> keyIter = selector.selectedKeys().iterator();
            while (keyIter.hasNext()) {
                SelectionKey key = keyIter.next();

                if (key.isAcceptable()) {
                    SocketChannel clientChannel = ((ServerSocketChannel) key.channel()).accept();
                    clientChannel.configureBlocking(false);
                    clientChannel.register(key.selector(), SelectionKey.OP_READ, ByteBuffer.allocate(32));
                }

                if (key.isReadable()) {
                    SocketChannel clientChannel = (SocketChannel) key.channel();
                    ByteBuffer buffer = (ByteBuffer) key.attachment();
                    long bytesRead = clientChannel.read(buffer);
                    // ...
                }

                keyIter.remove();
            }
        }
    }

既然Netty也肯定使用NIO，那么下面分析代码流程时也着重看一下Netty是在哪、如何使用NIO的API。

2.EventLoopGroup预准备

在主流程开始之前，EventLoopGroup构造方法里做了一些预准备的工作。

2.1 创建EventLoop组

NioEventLoopGroup继承自MultithreadEventLoopGroup和更上层的MultithreadEventExecutorGroup。其中，EventLoopGroup中指定使用的EventExecutor是NioEventLoop，而MultithreadEventLoopGroup指定了线程数(CPU数*2)和使用的线程工厂是DefaultThreadFactory。

注意：SelectorProvider.provider()始终返回第一次调用创建的SelectorProvider，所以这里调用provider()与后面NioServerSocketChannel中再次调用并不冲突。

// NioEventLoopGroup
    public NioEventLoopGroup(int nThreads, ThreadFactory threadFactory) {
        this(nThreads, threadFactory, SelectorProvider.provider());
    }

    @Override
    protected EventExecutor newChild(
            ThreadFactory threadFactory, Object... args) throws Exception {
        return new NioEventLoop(this, threadFactory, (SelectorProvider) args[0]);
    }

// MultithreadEventLoopGroup
    private static final int DEFAULT_EVENT_LOOP_THREADS;

    static {
        DEFAULT_EVENT_LOOP_THREADS = Math.max(1, SystemPropertyUtil.getInt(
                "io.netty.eventLoopThreads", Runtime.getRuntime().availableProcessors() * 2));
    }

    protected MultithreadEventLoopGroup(int nThreads, ThreadFactory threadFactory, Object... args) {
        super(nThreads == 0? DEFAULT_EVENT_LOOP_THREADS : nThreads, threadFactory, args);
    }

    @Override
    protected ThreadFactory newDefaultThreadFactory() {
        return new DefaultThreadFactory(getClass(), Thread.MAX_PRIORITY);
    }

利用这两个子类提供的信息，父类MultithreadEventExecutorGroup创建出NioEventLoop组和EventExecutorChooser。

// MultithreadEventExecutorGroup
protected MultithreadEventExecutorGroup(int nThreads, ThreadFactory threadFactory, Object... args) {
        if (threadFactory == null) {
            threadFactory = newDefaultThreadFactory();
        }

        children = new SingleThreadEventExecutor[nThreads];
        if (isPowerOfTwo(children.length)) {
            chooser = new PowerOfTwoEventExecutorChooser();
        } else {
            chooser = new GenericEventExecutorChooser();
        }

        for (int i = 0; i < nThreads; i ++) {
            try {
                children[i] = newChild(threadFactory, args);
            } catch (Exception e) {
                throw new IllegalStateException("failed to create a child event loop", e);
            }
        }
    }

2.2 EventLoop线程启动

NioEventLoop也是在构造方法中做了很多工作。它的父类SingleThreadEventExecutor会调用刚才NioEventLoopGroup中的线程工厂创建一个线程，并调用NioEventLoop覆写的run()方法。而run()方法中就是最为关键的事件循环代码，它对NioEventLoop构造方法创建的Selector不断的select()出就绪的事件。

// SingleThreadEventExecutor
    protected SingleThreadEventExecutor(
            EventExecutorGroup parent, ThreadFactory threadFactory, boolean addTaskWakesUp) {
        thread = threadFactory.newThread(new Runnable() {
            @Override
            public void run() {
                try {
                    SingleThreadEventExecutor.this.run();
                } catch (Throwable t) {
                    logger.warn("Unexpected exception from an event executor: ", t);
                }
            }
        });

        taskQueue = newTaskQueue();
    }

// NioEventLoop
    NioEventLoop(NioEventLoopGroup parent, ThreadFactory threadFactory, SelectorProvider selectorProvider) {
        super(parent, threadFactory, false);
        provider = selectorProvider;
        selector = openSelector();
    }

    private Selector openSelector() {
        final Selector selector;
        try {
            selector = provider.openSelector();
        } catch (IOException e) {
            throw new ChannelException("failed to open a new selector", e);
        }
        return selector;
    }

    @Override
    protected void run() {
        for (;;) {
            try {
                if (hasTasks()) {
                    selectNow();
                } else {
                    select(oldWakenUp);
                }
            } catch (Throwable t) {
                logger.warn("Unexpected exception in the selector loop.", t);
            }
        }
    }

3.ServerBootstrap主流程

粗看前面的Netty 4代码示例，好像看不出哪里是框架的起点。实际上，当我们调用bind()方法时，这就是整个Netty框架的起点。具体来说，可以分为三步：

创建Channel：创建NioServerSocketChannel以及底层NIO的Channel。

初始化Channel：初始化Channel和ChannelPipeline。

注册事件：绑定一个EventLoop到Channel上，并将Channel和关注的SelectionKey注册到Selector上。

绑定端口：绑定到某个监听端口上。

// AbstractBootstrap
    private ChannelFuture doBind(final SocketAddress localAddress) {
        final ChannelFuture regFuture = initAndRegister();
        final Channel channel = regFuture.channel();

        if (regFuture.isDone()) {
            ChannelPromise promise = channel.newPromise();
            doBind0(regFuture, channel, localAddress, promise);
            return promise;
        }
    }

    final ChannelFuture initAndRegister() {
        final Channel channel = channelFactory().newChannel();
        try {
            init(channel);
        } catch (Throwable t) {
            channel.unsafe().closeForcibly();
        }

        ChannelFuture regFuture = group().register(channel);
        return regFuture;
    }

3.1 创建Channel

ServerBoostrap根据我们传入channel()方法的NioServerSocketChannel.class，通过反射创建出Channel对象。注意：NioServerSocketChannel是Netty的包装类。真正的NIO Channel是在其构造方法中通过SelectorProvider创建的。

这里Netty没有用之前我们的NIO示例代码中的ServerSocketChannel.open()方法创建Channel，而是使用SelectorProvider。注释里写道是为了避免多个Channel同时创建时open()方法中的竞争条件。

// AbstractBootstrap
    public B channel(Class<? extends C> channelClass) {
        return channelFactory(new BootstrapChannelFactory<C>(channelClass));
    }

    private static final class BootstrapChannelFactory<T extends Channel> implements ChannelFactory<T> {
        @Override
        public T newChannel() {
            try {
                return clazz.newInstance();
            } catch (Throwable t) {
                throw new ChannelException("Unable to create Channel from class " + clazz, t);
            }
        }
    }

// NioServerSocketChannel
    private static final SelectorProvider DEFAULT_SELECTOR_PROVIDER = SelectorProvider.provider();

    public NioServerSocketChannel() {
        this(newSocket(DEFAULT_SELECTOR_PROVIDER));
    }

    private static ServerSocketChannel newSocket(SelectorProvider provider) {
        try {
            /**
             *  Use the SelectorProvider to open SocketChannel and so remove *  condition in SelectorProvider#provider() which is called by
             *  each ServerSocketChannel.open() otherwise.
             */
            return provider.openServerSocketChannel();
        } catch (IOException e) {
            throw new ChannelException(
                    "Failed to open a server socket.", e);
        }
    }

传入父类AbstractNioChannel的构造方法后，父类负责设置成了非阻塞模式。

// AbstractNioChannel
    protected AbstractNioChannel(Channel parent, SelectableChannel ch, int readInterestOp) {
        super(parent);
        this.ch = ch;
        this.readInterestOp = readInterestOp;
        try {
            ch.configureBlocking(false);
        } catch (IOException e) {
            throw new ChannelException("Failed to enter non-blocking mode.", e);
        }
    }

3.2 初始化Channel

创建完Channel后就可以为其做一些配置了。ServerBootstrap的init()方法会配置Channel的参数和属性，并创建ServerBootstrapAcceptor，它真正地持有workerGroup(childGroup)和我们定制的Handler。

// ServerBootstrap
    @Override
    void init(Channel channel) throws Exception {
        final Map<ChannelOption<?>, Object> options = options();
        synchronized (options) {
            channel.config().setOptions(options);
        }

        final Map<AttributeKey<?>, Object> attrs = attrs();
        synchronized (attrs) {
            for (Entry<AttributeKey<?>, Object> e: attrs.entrySet()) {
                AttributeKey<Object> key = (AttributeKey<Object>) e.getKey();
                channel.attr(key).set(e.getValue());
            }
        }

        ChannelPipeline p = channel.pipeline();
        if (handler() != null) {
            p.addLast(handler());
        }

        final EventLoopGroup currentChildGroup = childGroup;
        final ChannelHandler currentChildHandler = childHandler;
        final Entry<ChannelOption<?>, Object>[] currentChildOptions;
        final Entry<AttributeKey<?>, Object>[] currentChildAttrs;
        synchronized (childOptions) {
            currentChildOptions = childOptions.entrySet().toArray(newOptionArray(childOptions.size()));
        }
        synchronized (childAttrs) {
            currentChildAttrs = childAttrs.entrySet().toArray(newAttrArray(childAttrs.size()));
        }

        p.addLast(new ChannelInitializer<Channel>() {
            @Override
            public void initChannel(Channel ch) throws Exception {
                ch.pipeline().addLast(new ServerBootstrapAcceptor(
                        currentChildGroup, currentChildHandler, currentChildOptions, currentChildAttrs));
            }
        });
    }

默认情况下，ChannelPipeline里只有head和tail两个默认的Handler，tail是InBoundHandler，head是OutBoundHandler。真正完成主从Reactor交互的自然就是这里加入到Pipeline的ServerBootstrapAcceptor。

// AbstractChannel
    protected AbstractChannel(Channel parent) {
        this.parent = parent;
        unsafe = newUnsafe();
        pipeline = new DefaultChannelPipeline(this);
    }

// DefaultChannelPipeline
    public DefaultChannelPipeline(AbstractChannel channel) {
        this.channel = channel;

        tail = new TailContext(this);
        head = new HeadContext(this);

        head.next = tail;
        tail.prev = head;
    }

3.3 注册事件

完成了Channel和ChannelPipeline的初始化后，就要为Channel注册我们感兴趣的I/O事件了。尽管NIO的API很简单，但Netty中的注册流程还是比较复杂的：

以bossGroup的NioEventLoopGroup.register(Channel)方法为源头

经过由Chooser选取出的NioEventLoop的register(Channel)

最终才委托给Channel的unsafe().register(EventLoop)

首先，NioEventLoopGroup.register()方法会使用next()，借助EventExecutorChooser从EventExecutor数组中选出一个NioEventLoop，并调用其register()方法。

// MultithreadEventExecutorGroup
    @Override
    public ChannelFuture register(Channel channel) {
        return next().register(channel);
    }

// MultithreadEventExecutorGroup
    @Override
    public EventExecutor next() {
        return chooser.next();
    }

    private final class PowerOfTwoEventExecutorChooser implements EventExecutorChooser {
        @Override
        public EventExecutor next() {
            return children[childIndex.getAndIncrement() & children.length - 1];
        }
    }

NioEventLoop继承自SingleThreadEventLoop，它的register()方法会调用NioServerSocketChannel的unsafe工具进行注册。

// SingleThreadEventLoop
    @Override
    public ChannelFuture register(Channel channel) {
        return register(channel, new DefaultChannelPromise(channel, this));
    }

    @Override
    public ChannelFuture register(final Channel channel, final ChannelPromise promise) {
        channel.unsafe().register(this, promise);
        return promise;
    }

AbstractChannel中Unsafe匿名类会将传入的NioEventLoop绑定到当前Channel，最终触发doRegister()子方法完成注册工作。同时在注册完成后，Netty会向ChannelPipeline中发送channelRegistered和channelActive通知，这就是我们获得到的Channel通知的源头。

// AbstractChannel.AbstractUnsafe
    protected abstract class AbstractUnsafe implements Unsafe {
        /** true if the channel has never been registered, false otherwise */
        private boolean neverRegistered = true;

        @Override
        public final void register(EventLoop eventLoop, final ChannelPromise promise) {
            AbstractChannel.this.eventLoop = eventLoop;
            if (eventLoop.inEventLoop()) {
                register0(promise);
            }
        }

        private void register0(ChannelPromise promise) {
            try {
                boolean firstRegistration = neverRegistered;
                doRegister();
                neverRegistered = false;

                pipeline.fireChannelRegistered();

                if (firstRegistration && isActive()) {
                    pipeline.fireChannelActive();
                }
            } catch (Throwable t) {
                closeForcibly();
                closeFuture.setClosed();
            }
        }
    }

最终终于到了真正实现注册的地方：AbstractNioChannel.doRegister()会将底层JDK的ServerSocketChannel注册到当前绑定的eventLoop持有的Selector上。

// AbstractNioChannel
    @Override
    protected void doRegister() throws Exception {
        try {
            selectionKey = javaChannel().register(eventLoop().selector, 0, this);
            return;
        } catch (CancelledKeyException e) {
            // ...
        }
    }

3.4 端口绑定

绑定流程与注册类似，最终都是调用Channel的unsafe()工具类来完成。但区别是注册是从EventLoopGroup开始最终直接调用到Channel，而绑定是从Channel开始，经过了Pipeline中tail和head的处理才调用到Channel的。

// AbstractBootstrap
    private static void doBind0(
            final ChannelFuture regFuture, final Channel channel,
            final SocketAddress localAddress, final ChannelPromise promise) {

        channel.eventLoop().execute(new Runnable() {
            @Override
            public void run() {
                if (regFuture.isSuccess()) {
                    channel.bind(localAddress, promise).addListener(ChannelFutureListener.CLOSE_ON_FAILURE);
                }
            }
        });
    }

// AbstractChannelHandlerContext(TailContext)
    @Override
    public ChannelFuture bind(final SocketAddress localAddress, final ChannelPromise promise) {
        final AbstractChannelHandlerContext next = findContextOutbound();
        EventExecutor executor = next.executor();
        if (executor.inEventLoop()) {
            next.invokeBind(localAddress, promise);
        }
        return promise;
    }

// DefaultChannelPipeline.HeadContext
    static final class HeadContext extends AbstractChannelHandlerContext implements ChannelOutboundHandler {
        @Override
        public void bind(
                ChannelHandlerContext ctx, SocketAddress localAddress, ChannelPromise promise)
                throws Exception {
            unsafe.bind(localAddress, promise);
        }
    }

// AbstractChannel.AbstractUnsafe
    protected abstract class AbstractUnsafe implements Unsafe {
        @Override
        public final void bind(final SocketAddress localAddress, final ChannelPromise promise) {
            boolean wasActive = isActive();
            try {
                doBind(localAddress);
            } catch (Throwable t) {
                closeIfClosed();
                return;
            }
        }
    }

// NioServerSocketChannel
    @Override
    protected void doBind(SocketAddress localAddress) throws Exception {
        javaChannel().socket().bind(localAddress, config.getBacklog());
    }

4.总结梳理

至此，Netty服务就算是启动完毕，它已经开始监听端口上的请求了。现在就总结一下整个代码流程比较关键的地方。其实这一大片代码看下来，会发现ServerBootstrap和EventLoopGroup都是在互相配合，真正的核心是它们创建出NioEventLoop组和NioServerSocketChannel。每个NioEventLoop对应一个线程和一个Selector，NioServerSocketChannel会主动注册到某一个NioEventLoop的Selector上，NioEventLoop负责事件轮询。