Last week we had a new challenge. We wanted to collect some performance statistical data from our client side, for some enhancements we are making. And we wanted to make some reports on them. Normally, such kind of process requires the following steps:
- Collecting the data.
- Sending them to you servers.
- Aggregating the data
- Creating reports.
Collecting the data depends on your client type and which data you need. And, depending on your database, there are many techniques to aggregate and create reports.
But how to send data to your server?
The intuitive way is to create new servers, behind a load balance, with a monitoring system to control the number of them. Sound a lot of work to accomplish a relatively small task.
The solution that we came up with is to use a CDN to do all that work for us. Not by posting data to the CDN, but rather to serialize the data into query string and to retrieve 1×1 image from the CDN and appending the query string to the image url. Then with a simple process, we download the log files of the CDN provider, parse them, and extract the data back from those query strings.
To accomplish that, all you need is a reliable CDN provider and a durable storage to store the logs. And you get the following benefits:
- No need to setup servers or background processes
- No need to worry about scalability
- Very simple to accomplish
- No need to store your raw data. That kept in the log files.
- As most of the CDN providers charges based on used bandwidth, downloading a small image is relatively cheap.
We even managed to expand the solution to be extendable and generic. Now it’s very easy for us and for the management to make a decision regarding future similar processes.