IOCP完成端口介绍及完成端口内部实现

HANDLE CreateIoCompletionPort(
HANDLE FileHandle,
HANDLE ExistingCompletionPort,
DWORD CompletionKey,
DWORD NumberOfConcurrentThreads
);

BOOL GetQueuedCompletionStatus(
HANDLE CompletionPort,
LPDWORD lpNumberOfBytesTransferred,
LPDWORD CompletionKey,
LPOVERLAPPED *lpOverlapped,
DWORD dwMiillisecondTimeout
);

CompletionKey = handle 的 ID（用户自定义）

IpOverlapped = 重叠结构（一个重叠结构对应一个IO操作）

struct PerIoData {

    OVERLAPPED overlapped;

    SOCKET sockAccept;

    OperationType op;

    char buff [ BUFF_SIZE ];

};

完成端口主要就是一个IO完成包队列，但其内部实现还涉及到一个（LIFO的）工作线程等待队列，完成端口的首地址就是完成包队列的首地址。

当CreateIoCompletionPort被调用来创建一个完成端口时，一个IO完成包队列被创建出来，这个队列的成员变量中有一个最大同步线程数量

当CreateIoCompletionPort被调用来把一个handle绑定到完成端口上时，函数NtSetInformationFile会被调用，这个函数的工作是：

（1）建一个_IO_COMPLETION_CONTEXT结构体，把里面的Port指向IOCP，Key赋值为从CreateIoCompletionPort传入的CompletionKey参数

（2）把handle中的成员变量CompletionContext指向这个_IO_COMPLETION_CONTEXT结构体

完成绑定后，当绑定的IO操作完成后，Windows的IO系统内部的IO管理函数IopCompleteRequest会被执行，这个函数会检查handle是否被绑定到完成端口上，如果是，IO系统会调用函数KeInsertQueue把完成包放到完成端口的完成包队列上（通过那个_IO_COMPLETION_CONTEXT结构体的Port指针），KeInsertQueue会把等待在一个queue上的线程唤醒（如果有线程在等待），此处的queue就是完成包队列。

当GetQueuedCompletionStatus被线程调用来获取完成包时，API函数NtRemoveIoCompletion被调用，（检查参数和获取完成端口指针后）接着调用KeRemoveQueue

在内部实现中，完成端口内部的工作线程等待队列会维持活动线程的计数和一个最大活动线程数量，如果当前活动线程数>=最大活动线程数，当前线程会被阻塞，加入到等待线程队列中等待被唤醒。

Completion Port Internals

A call to the Win32 API CreateIoCompletionPort with a NULL completion port handle results in the execution of the native API functionNtCreateIoCompletion, which invokes the corresponding kernel-mode system service of the
same name. Internally, completion ports are based on an undocumented executive synchronization object called a Queue. Thus, the system service creates a completion port object and initializes a queue object in the port's allocated memory (a pointer to the
port also points to the queue object since the queue is at the start of the port memory). A queue object has (coincidentally) a concurrency value that is specified when a thread initializes one, and in this case the value that is used is the one that was passed
toCreateIoCompletionPort. KeInitializeQueue is the function thatNtCreateIoCompletion calls to initialize a port's queue object.

When an application calls CreateIoCompletionPort to associate a file handle with a port the Win32 API invokes the native functionNtSetInformationFile with the file handle as the primary parameter. The information class that
is set isFileCompletionInformation and the completion port's handle and theCompletionKey parameter fromCreateIoCompletionPort are the data values.NtSetInformationFile dereferences the file handle
to obtain the file object and allocates a completion context data structure, which is defined in NTDDK.H as:

typedef struct _IO_COMPLETION_CONTEXT {
PVOID Port;
ULONG Key;
} IO_COMPLETION_CONTEXT, *PIO_COMPLETION_CONTEXT;

Finally, NtSetInformationFile sets the CompletionContext field in the file object to point at the context structure. When an I/O operation completes on a file object the internal I/O manager functionIopCompleteRequest
executes and, if the I/O was asynchronous, checks to see if theCompletionContext field in the file object is non-NULL. If its non-NULL the I/O Manager allocates a completion packet and queues it to the completion port by callingKeInsertQueue
with the port as the queue on which to insert the packet (remember that the completion port object and queue object are synonymous).

When GetQueuedCompletionStatus is invoked by a server thread, it calls the native API functionNtRemoveIoCompletion, which transfers control to theNtRemoveIoCompletion system service. After validating parameters
and translating the completion port handle to a pointer to the port,NtRemoveIoCompletion callsKeRemoveQueue.

As you can see, KeRemoveQueue and KeInsertQueue are the engine behind completion ports and are the functions that determine whether a thread waiting for an I/O completion packet should be activated or not. Internally, a
queue object maintains a count of the current number of active threads and the maximum active threads. If the current number equals or exceeds the maximum when a thread calls KeRemoveQueue, the thread will be put (in LIFO order) onto a list of threads waiting
for a turn to process a completion packet. The list of threads hangs off the queue object. A thread's control block data structure has a pointer in it that references the queue object of a queue that it is associated with; if the pointer is NULL then the thread
is not associated with a queue.

So how does NT keep track of threads that become inactive because they block on something other than the completion port? The answer lies in the queue pointer in a thread's control block. The scheduler routines that are executed in response to a thread blocking
(KeWaitForSingleObject, KeDelayExecutionThread, etc.) check the thread's queue pointer and if its not NULL they will callKiActivateWaiterQueue, a queue-related function.KiActivateWaiterQueue
decrements the count of active threads associated with the queue, and if the result is less than the maximum and there is at least one completion packet in the queue then the thread at the front of the queue's thread list is woken and given the oldest packet.
Conversely, whenever a thread that is associated with a queue wakes up after blocking the scheduler executes the functionKiUnwaitThread, which increments the queue's active count.

Finally, the PostQueuedCompletionStatus Win32 API calls upon the native functionNtSetIoCompletion. As with the other native APIs in the completion port group, this one invokes a system service bearing the same name, which
simply inserts that packet onto the completion port's queue usingKeInsertQueue.