dracoblue.net

Fast clicktracking with nginx logs

When your web site is static (generated upfront or cached in varnish/nginx), you cannot update view/hits/download counters on each request. This click tracking needs to be done with additional requests.

Let's say our download urls look like this:

http://example.org/download-file/craur/craur-1.2.0.zip

We add a new location to our nginx file:

location ~ ^/download-file/[^/]+/([^/]+)$ {
    access_log /home/example/log/example.org.downloads.log short;
    alias /home/dracoblue/files/$1;
}

The short access_log format is custom and needs to be defined in the nginx.conf:

log_format short  '[$time_local] $status $request_uri';

If you make this, the log file at `` will look like this:

[07/Dec/2013:11:56:09 +0100] 200 /download-file/craur/craur-1.2.0.zip
[07/Dec/2013:12:02:51 +0100] 200 /download-file/naith/naith-0.1.0.zip
[07/Dec/2013:12:02:59 +0100] 200 /download-file/craur/craur-1.2.0.zip

Now we build a bash command line, to get the total amounts of all downloads in this timeframe:

cat example.org.downloads.log \
  | grep "200 " \
  | cut -f '4-' -d '/' \
  | sort | uniq -c
  • cat example.org.downloads.log reads the file.
  • grep "200 " outputs only 200 reponses.
  • cut -f '4-' -d '/' removes everything except /download-file/ ... .zip
  • sort | uniq -c sorts everything and afterwards counts it with uniq.

The final output looks like this:

  55 download-file/craur/craur-1.2.0.zip
   2 download-file/naith/naith-0.1.0.zip

The is very easy to parse with your favorite programming language. Then you write the "hits" for those files into the database and you are done.

Updated: 2014/01/05 added Charles recommendation for log import:

When importing the file, it's important that you send a USR1 signal to the nginx process, to ensure that the nginx process re-opens the log files after the move:

mv /home/example/log/example.org.downloads.log /home/example/log/example.org.downloads.log.0
kill -USR1 `cat /var/run/nginx.pid`

This will allow nginx to continue writing in the file.

If your site makes lots of traffic, you might execute this cron job more often.

In linux, nginx, open source, ubuntu by @ 13 Dec 2013

comments powered by Disqus

Recent Files

Advertisement

Recent Dev-Articles

Read recently

About

Blogroll