dracoblue.net

Fast clicktracking with nginx logs

When your web site is static (generated upfront or cached in varnish/nginx), you cannot update view/hits/download counters on each request. This click tracking needs to be done with additional requests.

Let's say our download urls look like this:

http://example.org/download-file/craur/craur-1.2.0.zip

We add a new location to our nginx file:

location ~ ^/download-file/[^/]+/([^/]+)$ {
    access_log /home/example/log/example.org.downloads.log short;
    alias /home/dracoblue/files/$1;
}

The short access_log format is custom and needs to be defined in the nginx.conf:

log_format short  '[$time_local] $status $request_uri';

If you make this, the log file at `` will look like this:

[07/Dec/2013:11:56:09 +0100] 200 /download-file/craur/craur-1.2.0.zip
[07/Dec/2013:12:02:51 +0100] 200 /download-file/naith/naith-0.1.0.zip
[07/Dec/2013:12:02:59 +0100] 200 /download-file/craur/craur-1.2.0.zip

Now we build a bash command line, to get the total amounts of all downloads in this timeframe:

cat example.org.downloads.log \
  | grep "200 " \
  | cut -f '4-' -d '/' \
  | sort | uniq -c
  • cat example.org.downloads.log reads the file.
  • grep "200 " outputs only 200 reponses.
  • cut -f '4-' -d '/' removes everything except /download-file/ ... .zip
  • sort | uniq -c sorts everything and afterwards counts it with uniq.

The final output looks like this:

  55 download-file/craur/craur-1.2.0.zip
   2 download-file/naith/naith-0.1.0.zip

The is very easy to parse with your favorite programming language. Then you write the "hits" for those files into the database and you are done.

Updated: 2014/01/05 added Charles recommendation for log import:

When importing the file, it's important that you send a USR1 signal to the nginx process, to ensure that the nginx process re-opens the log files after the move:

mv /home/example/log/example.org.downloads.log /home/example/log/example.org.downloads.log.0
kill -USR1 `cat /var/run/nginx.pid`

This will allow nginx to continue writing in the file.

If your site makes lots of traffic, you might execute this cron job more often.

In linux, nginx, open source, ubuntu by
@ 13 Dec 2013, Comments at Reddit & Hackernews

Give something back

Were my blog posts useful to you? If you want to give back, support one of these charities, too!

Report hate in social media Campact e.V. With our technology and your help, we protect the oceans from plastic waste. Gesellschaft fur Freiheitsrechte e. V. The civil eye in the mediterranean

Recent Dev-Articles

Read recently

Recent Files

About