Queue of tasks or my own solution for Thundering Herd?

Question

I am developing a network server application in C++ and came across the problem Thundering Herd, once I designed to have multiple accepting threads (accept()) customers' connections simultaneously.

By my searches, the most common solution is to do a Queue of FIFO tasks, which consists of pointers to functions that are being executed by threads. In these functions already comes the socket FD of the accepted client connection, then the accept() is being solely responsible for the main process, without Thundering Herd here.

But I think it’s kind of complicated to implement that and I had another idea, which is this::

Thread:

• has an ID;
• monitors the server/listening socket for new client connections using kqueue, epoll, etc.
• when a new connection arrives, it prompts the Threads Controller the ID of the thread that should accept this connection. If the ID received is the ID of the thread itself:
  1. accepted (accept()) new connection;
  2. tells the Threads Controller to change the ID of the thread that should accept the connection to the ID of the next thread;
  3. tells the Threads Controller who is now busy;
  4. receives client requests in socket and processes responses;
  5. tells the Threads Controller which is now free to meet another connection;

Threads Controller:

• keeps the thread id you should accept (accept()) the next connection;
• maintains an atomic access vector (std::atomic<std::vector>) of structs which contains the thread ID and if it is busy (at the end is similar to a std::map);
• when the Thread performs part 2 above, the Threads Controller traverse the array, get the next thread that is not busy and set its ID to the ID of the thread that you should accept (accept()) the next connection.

Do you think this solution is better than the task question? In both there is a time when only 1 thread does something at a time (get the next task in Ueue and accept the new connection), in both there is the use of std::atomic, both seem to solve the problem...

In my view there are no problems in my solution, but since I’m not very experienced programming network applications, multithreading, etc., there may be something I’m missing, something that can be a bottleneck, that can lock a thread and stop others from continuing, I don’t know.

If there are no problems, this can be a much easier solution to implement for those who face the most Thundering Herd. There is no way I can do a test application here and simulate a real environment, since my only computer is an old and weak single core.

What do you say? I thank you for your attention.

Answer 1

From what I understand, according to your solution, when a connection arrives all threads wake up, and the one that has the ID provided by the controller accepts the connection. If that is the case, isn’t this just a case of Thundering Erd? That is, whenever a connection arrives, all threads wake up, even if only one will accept the connection.

On the other hand, in a task system where each thread has its own task queue, when a connection arrives, the scheduler (who plays the role of what you call the controller) places this task in one of the queues, and only the thread responsible for that queue wakes up. I believe you get better performance by following this strategy.