When developing a large application, it is sometimes necessary to split off well-defined subtasks so that they can be run on their own servers.
One example already developed in FieldMotion is our PDF microservice. As we started to grow, it became obvious that PDF generation was a bottle-neck – it takes a while to generate a PDF, and while that is happening, it slows down anything else that happens on that server. The solution here was to move PDF generation off to a separate server dedicated to that purpose.
In that case, we used an asynchronous queueing system. We will use a simpler synchronous system for this article.
The problem I’m currently solving is that …well, it doesn’t really matter what problem we’re solving, really. What matters is the details of how to handle authentication, receive the request, and send a reply.
How does a microservice work?
In our synchronous solution, the short answer is that we accept a request, and return a response.
The request must have some form of authentication and include details of the action to be performed.
The response should include a status such as whether or not the action was successful, and whatever errors were encountered along the way.
The authentication method we will use is called HMAC, which is defined simply as hash(key || hash(key || message)), where “hash” is a hashing algorithm (md5, sha1, etc.), “key” is a shared secret key, and “message” is the request, converted into a string. The double hash might seem a bit pointless, but apparently there are security issues with single-hashing, but no-one has yet found an issue with double-hashing.
Don’t allow the same request to be run more than once. If a request is repeated, it is possibly someone trying to hack. To stop repeated requests, in the microservice server, simply keep a note of all hashes that have been seen, and check them before running the message’s request. So if a server sends a request, and then sends the exact same request again, the microservice simply checks its notes, sees that it has already noted that MD5, and can safely ignore it.
With a busy server, this list of hashes can get large quickly, so it’s good to clear out any old hashes every now and then to keep the list small.
But then how do you stop someone from repeating the same request in five minutes that they just sent? Simply add the current time to the message before hashing. On the microservice, if the time in the message is more than 60 seconds ago (for example), then ignore it, as it’s probably a repeat. and if it’s not a repeat? 60 seconds to send a request between two servers in the same datacentre? You have other problems…
There is one further question that needs to be answered. Let’s say that someone manages to hack the requesting server and gets a copy of the secret key? What do we do? Let’s say we have 50 other servers that use that same microservice. We can’t afford to simply change the key on the microservice server, as that will cause all 50 of the servers to suddenly stop working!
The answer here is to have a list of secret keys on the microservice server. This way you can simply disable the one that’s been compromised, and the rest will still work. Just give a different shared key to each server, and a small identifying key name so the microservice knows which key to check against.
So, the sequence we now have is
On Requesting Server
- Generate a message that describes the request.
- Add current time in seconds to the message.
- Generate a hash by running HMAC with the message and the secret key.
- Send message, hash, and key name to microservice.
On Microservice Server
- If time in message is more than 60 seconds ago, fail.
- Check local notes of recently run hashes. If the received hash is in the list, fail.
- If the secret key identified by the received key name doesn’t exist, fail.
- Generate a hash by running HMAC with the message you received and the secret key identified in step 3.
- If the generated hash is not exactly the same as the received hash, fail.
- Note the hash so it is not run again.
- Decode the message and run the embedded request.
- Return the result of the action to the requesting server.
Creating a micro-service server using the queuing method that I mentioned for the PDF solution is more complicated. I’ll write about that kind of server next time it needs to be done here.