10 Million Websockets? More Like 10 Million Problems (Solved the Hard Way)

So, the initial idea was simple: thread-per-connection. Sounds good on paper, right? Wrong. Horribly, painfully wrong. I deployed that first version using Python's default threading, and within minutes – I'm talking *minutes* – the whole thing choked. The server, a shiny new AWS t3.xlarge, just gasped and died. Turns out, creating and managing 10 million threads is… a bit much. CPU usage went through the roof, memory leaked like a sieve, and context switching ate us alive. It was a garbage fire. Don't even get me started on debugging that mess. I went back to the drawing board.

Next up, asyncio. Everyone's favorite buzzword, right? Okay, it's *better* than threads, I'll give it that. But it wasn't a silver bullet. I migrated everything to asyncio, thinking I was so smart. Initial tests looked promising... until the load hit about 1 million connections. Then, I started seeing weird delays, dropped messages, and a general sense of impending doom. It turns out that even with asyncio, you can still overwhelm a single process. And that Global Interpreter Lock (GIL) in Python? Still a pain. I spent a week profiling code, trying to figure out where the bottleneck was. The problem? Single core usage maxed out. I needed to distribute the load.

So, I thought, 'Okay, more processes!'. I split the websocket handling across multiple Python processes using multiprocessing. Queue-based inter-process communication, the whole shebang. I even used Redis as a broker thinking this would good. And guess what? It worked! Sort of. Until it didn't. The overhead of constantly passing messages between processes added latency. Redis became a bottleneck. And debugging a distributed system with a million connections across multiple processes?.I swear I aged five years in that week. The monitoring dashboards lit up like a Christmas tree of failure. Each problem was a new cascading failure. I needed something better. Simpler.

Fine, I said. If I can't handle the connections efficiently in Python, I'll offload it. Nginx to the rescue! I configured Nginx to handle the websocket connections and proxy the messages to my backend processes. This helped with connection management and load balancing. It was *better*. I started seeing stable connection counts and better CPU utilization, but the latency was still bugging me. The problem? Nginx isn't magic. It still has to proxy those messages, and that adds overhead. Plus, configuring Nginx for millions of websockets is… an art. A dark art involving tweaking obscure kernel parameters and praying to the networking gods. I got close, but it wasn't *quite* there. I still had a sneaking suspicion the core Python processing was dragging me down.

Alright, time for the really dirty tricks. I went back to the multi-processing approach, but this time, I pinned each process to a specific CPU core. I used `taskset` to do this. The idea was to maximize cache locality and minimize context switching. I also optimized the message processing code like a maniac, focusing on reducing memory allocations and unnecessary data copies. It was painful. It was ugly. But it worked. Kind of. I still had to carefully tune the number of processes to match the number of cores, and I had to monitor the system like a hawk. But finally, *finally*, I got 10 million websocket connections running on a single server. Was it elegant? Would I recommend it? Only if yourself. But it got the job done. Now, if you'll excuse me, I need a drink.