Setting up analytics - technology choices

Hi,

I am in the process of setting up my own little analytics database for generating some technical indicators as well as for options Greek. I wrote the initial technical analysis part in Perl which I found was too slow when more than one day needs to be processed. So re-wrote it into Octave (clone of Mat lab) which was faster but querying the result from Octave native files was a pain. So yet again decided to use a database system to keep the data and chose kdb (trail version for now). However Octave does not seem to have any interface to kdb, once again I had to go back to Perl and write it for kdb - which is now fast enough.

As you may realize Perl is not really handy for modeling ( while Octave is ), so although I am done for now I am realizing this is not perfect for future expansion considering that I have not yet started with the Options pieces. I was considering the "q" language which works with kdb as a script which seem to be quite different from any other scripting languages I have used in the past and not as much readable. I am sure a few of you out there have tacked similar problem as me with the wide choice of languages out there. Any suggestions which one of these (or something not mentioned here) could be a winner long term?

Tx

For doing the same tasks that you're doing I use C# and SQL Server. The C# compiler is free and comes with any .Net installation. If you need an editor for the language there are free Express versions. Also SQLServer Express is free with a limit of 1 processor and 2 GB databases.

I take the DB latency out of the equation by doing this type of testing on a 64bit system with plenty of memory so I read everything I need out of the DB and into memory. I also find that using multiple threads really speeds things up if you have a multi-core system because many of these analytics tasks can be parallelized. The latest version of C# (version 4) has a very easy way to parallelize loops which is what you're usually doing a lot of when you're doing this type of work.

I hope that helps.