Using a profiler to find that one line change for 10x Speedup

For a few years now we've been working on an explorer for the TON Blockchain called tonscan.com. The explorer indexes everything that happens on the blockchain, and in particular tokens - our name for special smart contracts that need more indexing than just a wallet. This can be NFTs, or things like trading pools for distributed exchanges. TON has some 50-150 txs per second - many of these touch one of these tokens - and we queue that token for needing an update. Another process will consume this queue and do the actual update - this typically means calling a get method on that contract, decoding the binary result and storing the result in the database.

For months I've half-arsed tried to optimise the token updater for Tonscan.

My token updater does this at around 10-20 tokens per second, which used to be barely enough, but isn't any more. So the queue of pending token updates kept growing. So I finally sat down with an actual profiler - and the reason it was slow was super-super stupid. We use a little microservice for taking the code+data for a token and running the get-method, essentially running our own copy of the TON virtual machine.

I hit this microservice via HTTP. I had previously scaled up the number of instances for this microservice, but it didn't change much, so I figured the bottleneck was elsewhere. But the profiler told me most of my time was spent in sslcontext.load_verify_locations 🤔

So we use httpx, and I had totally ignored re-using a client, so every inner look of my process did:

async with httpx.AsyncClient(): ...

Although my microservice runs internally and uses plain HTTP, this still would read all SSL certs it should trust from disc. And this really took the most time. Once I added verify=False to that line, I got a 10x speedup. Now we do 200 tokens per second 😭 (I now also refactored it to actually REUSE the client, which also gives you connection pooling) I knew this line was sloppy, but I figured, "how bad can it be, there must be much bigger issues elsewhere" ...

Now you know - please re-use your HTTP clients! (And also please use a profiler, don't trust your intuition!)