Tag: clojure

General

Favorite languages, why so great? and why not so much?

Posted on by devender

About my favorite languages, I actually have 2 favorite languages

  • Ruby: for all scripting and making quick apps.
  • Clojure: for development.

Why Ruby is great

  1. The language was designed for programmer use, you can see that from the api which is totally intuitive.
  2. Lots of libraries, my favorite is Sinatra which lets you build quick and dirty web apps and the other is Sequel.
  3. I wrote a blog post on how to delete RFC-822 in compatible emails (if you are a developer using linux and your company uses Outlook you know what I am talking about), this is a simple example of how I have used Ruby to make quick and dirty scripts.

I have used Ruby numerous times to write scripts to fix production data, correct files, and to generate complex reports. I have used Sinatra with Google Charts to make web apps that can show load times, server status ….

Why Ruby is not so great

  1. Not really meant for performance, recent years there is a push to develop a virtual machine for Ruby but it is still not anywhere close to C/Java performance.
  2. Rails is a pain to deploy, Heroku takes away the pain but what do you do if you have to deploy internally ? I personally have 2 apps on Heroku one of which is http://first3links.com/

Why Clojure is great
I have been on a quest to learn a functional programming language for the past 3 years, I have read the Erlang book (please see the various posts I wrote about Erlang here). Erlang is a fine language but I lost interest in it after I could not find a single good library that can connect Erlang to Oracle. The problem, there are too few 3rd party libraries. The next language I looked at was Haskell, lots of libraries and seems to be good at performance on the surface, problem I see is acceptance by business, where most of the code is in Java. Then I found Clojure and fell in love with it.

  1. It is just another DSL for the JVM, if you provide type hints the code generated will be the same as what Java would (can easily sneak it in).
  2. Totally embraces the JVM unlike JRuby.
  3. The author Rich Hickey has done a lot to reduce the pain points of lisp.
  4. Finally a language that frees you mind of OOP ( Have you ever noticed how much time you spend in trying to achieve the best object model when a simple one would do ? and for what ? the customers don’t care as long as it works, the computers sure don’t care as long it is 0s and 1s)
  5. Code is so concise and elegant.

Why Clojure is not great.

  1. It has been called as the language with the steepest learning curve on the JVM, I tend to agree with it.
  2. Unlike Scala you have no wiggle room, it is either functional code or nothing ( I like this feature actually).
  3. Debugging is a major pain point. (Though there has been improvement with the latest clojure-swank).

I have written many posts on Clojure on my blog you can see them here. In the most recent post I show you one can parse a one million record file in less than 15 seconds with clojure.

0
General

Denormalizing One million records with Clojure.

Posted on by devender

MovieLens is a research project that provides datasets of various sizes and attributes, containing movie ratings. These datasets are free to download and use for non-commercial purposes. They have done an awesome job putting this data together and a big thanks goes to them for making it available.

I wanted to exercise my Clojure skills (more like add to my tiny set of Clojure skills 🙂 ) and it just so happens that I recently came across the MovieLens project, so how about analyzing that data using Clojure ?

One of the datasets they make available is the One Million Dataset, this set consists of 3 files

  1. “movies.dat” containing 3883 movie listings, contains title, genre…
  2. “users.dat” containing 6040 unique users, contains age, occupation, gender …
  3. “ratings.dat” containing 1000209 movie ratings, that references movie id and user id from the above 2 files.

I could analyze this data to answer questions such as, What age group gave the most ratings ? or What was the highest rated movie for a given time period ?

But before I could do this I wanted to denormalize the ratings file so that it also contains the user and movie information, why ? cause I don’t want to look it up when I am analyzing the data, each record should be self contained.

The outline of the program is quite simple

  • Read the users file into memory
  • Read the movies files into memory
  • For each line in the ratings
    • Find the corresponding movie and user
    • Print it out to a file.

Take a minute to think how would you do this in java and then look at the below code. I ran it on a Dell laptop dual 2.2Ghz laptop with 4 gig of ram and care to guess how long it takes ?? scroll down for answer.
–
–

(ns com.dev.file-reader
 (:use [clojure.contrib.duck-streams])
 (:import [java.io BufferedReader FileReader BufferedWriter FileWriter]))

(defstruct user :id :gender :age :ccupation :zip-code)
(defstruct movie :id :title :genres)

(defn format-user [user] (str (:id user) "::" (:gender user) "::" (:age user) "::" (:ccupation user) "::" (:zip-code user)))

(defn format-movie [movie] (str (:id movie) "::" (:title movie) "::" (:genres movie)))

(defn read-user-file [fileName]
 (loop [users {} fileSeq (read-lines fileName)]
   (let [line (first fileSeq)]
     (if (nil? line)
     users
     (let [tokens (.split line "::")
           id (aget tokens 0)
           user (struct user id (aget tokens 1) (aget tokens 2) (aget tokens 3) (aget tokens 4))]
        (recur (merge users {id user}) (rest fileS)))))))

(defn read-movies-file [fileName]
 (loop [movies {} fileSeq (read-lines fileName)]
   (let [line (first fileSeq)]
     (if (nil? line)
     movies
     (let [tokens (.split line "::")
           id (aget tokens 0)
           movie (struct movie (Integer/parseInt (aget tokens 0)) (aget tokens 1) (aget tokens 2))]
         (recur (merge movies {id movie}) (rest fileS)))))))

(defn convert-ratings-file
 "read the ratings file and denormalize it"
 [moviesF usersF ratingsF outputF]
   (let [movies (read-movies-file moviesF) users (read-user-file usersF)]
     (with-open [#^BufferedReader rdr (BufferedReader. (FileReader. ratingsF) 1048576)
                 #^BufferedWriter wtr (BufferedWriter. (FileWriter. outputF) 1048576)]
       (doseq [line (line-seq rdr)]
         (let [tokens (.split line "::")
               user-id (aget tokens 0)
               movie-id (aget tokens 1)
               user (get users user-id)
               movie (get movies movie-id)
               rating (aget tokens 2)
               timestamp (aget tokens 3)]
 (.write wtr (str (format-user user) "::" (format-movie movie) "::" rating "::" timestamp "\n")))))))

(defn doIt []
 (time (convert-ratings-file
 "movielens-1m/movies.dat"
 "movielens-1m/users.dat"
 "movielens-1m/ratings.dat"
 "movielens-1m/output.dat"
 )))

So ready with you guess ??
I ran the program 5 times and here is the output

"Elapsed time: 12130.035819 msecs"
"Elapsed time: 13113.92823 msecs"
"Elapsed time: 13364.234216 msecs"
"Elapsed time: 12553.478168 msecs"
"Elapsed time: 14488.706176 msecs"

On average 13.130076521799994 Seconds to read in 1 million records, for each record look up the movie and user and write it back to the disk.

Clojure puts the FUNctional back in programming.

3
General

Redis and Clojure

Posted on by devender

Check out my previous post about Redis.

In this post I build a very simple example of using Redis with Clojure. I will be using a client library for Redis written in Clojure called redis-clojure. You could also use the java library, to see complete list of supported languages go to this link.

So here we go..

  1. Create a simple clojure project (I personally use Leinigen), to create a new project execute ‘lein new com.dev/try-redis‘ this will create an entire project structure.
  2. Edit the project.clj file under the newly created project directory and add a new dependency for redis-clojure, the file should look close to this after you are done. 
    (defproject com.dev/try-redis "1.0.0-SNAPSHOT"
  :description "simple example of using redis"
  :dependencies [[org.clojure/clojure "1.1.0"]
                 [org.clojure/clojure-contrib "1.1.0"]
                 [redis-clojure "1.0.3-SNAPSHOT"]]
  :dev-dependencies [[swank-clojure "1.2.1"]])
  3. run ‘lein deps‘ so that all the dependencies are downloaded.
  4. Edit the file core.clj under the directory try-redis/src/com/dev/try_redis, and add the following. 
    (ns com.dev.try-redis.core
  (:require redis))

(defn test-redis []
     (redis/with-server {:host "127.0.0.1" :port 6379 :db 0}
       (do
         (redis/set "foo" "bar")
         (println (redis/get "foo")))))

    On lines 7 and 8 we are setting key value pair and retriving the value.

  5. Start the redis server ‘./redis-server redis.conf‘
  6. Now we are ready to execute the script, there are 2 ways to do this.
    1. The easiest way is just going to your porject root directory and run ‘lein repl‘ (see the below oouput) which opens a read evaluate loop and once you have that run ‘(load-file “src/com/dev/try_redis/core.clj”)‘ to load the file and then you can run ‘(com.dev.try-redis.core/test-redis)‘ to run the example.
    2. I personally use emacs/slime, but for this option you need to have emacs and slime-clojure installed (See my emacs page). Run ‘lein swank‘ in the project directory and then in your emacs connect to it using ‘M-x slime-connect‘, this will open up a repl, do a C-c C-k to compile the file and in the repl you can execute using ‘(com.dev.try-redis.core/test-redis)’

If everything has gone will you should see this output.

Clojure 1.1.0
user=> (load-file "src/com/dev/try_redis/core.clj")
#'com.dev.try-redis.core/test-redis
user=> (com.dev.try-redis.core/test-redis)         
bar
nil
user=>
2
General, Tech

Clojure, JDBC and using Dates.

Posted on by devender

I did not find many examples of using dates with Clojure/JDBC, after some initial struggle I got an example working. So here it is, hopefully it helps someone. By the way this page is a good source if you are interested in Clojure/JDBC.

Source File

(ns clojure-bi.simple
  (use clojure.contrib.sql))

(defn connection-props [connect-string user password]
  {:classname "oracle.jdbc.driver.OracleDriver"
	 :subprotocol "oracle:thin"
	 :subname (str "@" connect-string)
	 :user user
	 :password password })

(defn dev-db-props [user password]
  (connection-props "devdb.db.cj.com:1521:devdb" user password))

(defn make-sql-date [year month day]
  (java.sql.Date. 
   (.getTimeInMillis 
    (java.util.GregorianCalendar. year month day))))

(defn get-company-name [db id]
  (with-connection db
    (with-query-results rs ["select organization from company where id = ? " id] 
                  (first rs))))

(defn find-by-date [db date]
  (with-connection db
    (with-query-results rs ["select organization from company where trunc(Date_) = ? " date]
      (first rs))))

Test file

(ns clojure-bi.simple-test
  (:use [clojure-bi.simple] :reload-all)
  (:use [clojure.test])
  (import (java.util Date)))

(def dev-db (dev-db-props "devender" "test"))

(def expected-company {:organization "Client Services"})

(deftest get-company-name-test
  (is (= (get-company-name dev-db 2) expected-company)))

(def date-to-use (make-sql-date 1999 6 15))

(deftest find-by-date-test
  (is (= (find-by-date dev-db date-to-use) expected-company)))

The interesting parts are the find-by-date function in the source and the test function find-by-date-test, you have to pass in a java.util.sql Date.

1
General

Nice Presentation on F#

Posted on by devender

If you are studying Clojure or for that matter Scala or any other functional programming language I highly recommend that you check out this presentation “Introduction to Microsoft F#” You can follow along and try out his examples in your own language, the talk is funny with a lots of information that could apply to any functional language. I was really impressed by the F# ide, man I wish Clojure has the same level of IDE support.

1
General

Loving Clojure

Posted on by devender

I seem to be liking Clojure …

Let me backup a bit, in the last couple of weeks I have been debating between picking up Scala or Clojure (don’t get me wrong Ruby is still my favorite).

I always wanted to pick up a functional programming language so I dabbled a bit with Erlang and Haskell, liked Haskell a lot but without much practice it kind of died (sad times)  and Scala seems too much like Java, yeah I know it seems to have a bigger crowd than Clojure and there are a lot of big names behind it.

Maybe that’s exactly why I choose Clojure (since its the underdog), or cause it is different enough from Java or simply cause it has a better syntax and seems more elegant (apparently Clojure has better integration with Java, don’t quote me on it), anyways I decided to learn Clojure.

Peepcode has a nice screen-cast to get you started off on Clojure. If you are on the Mac there is a nice bundle for TextMate and anywhere else Netbeans with the enclojure plugin seems to be the best.

On a side note it seems more and more that Netbeans has the latest and greatest plugins for everything, then comes IntelliJ and finally eclipse, what’s going on with eclipse ? has it reached its peak and now it will start dropping off ? but on the flip side there seems to be more and more apps built on top of the Eclipse RCP like Xmind, so is Eclipse no longer going to be the leader of the IDE and just become a platform for building RCPs. This of course depends  on what Oracle is going to do with NetBeans, I really hope they give the same amount of love to NetBeans as Sun did.

Ok getting back to Clojure, don’t get your panties in a bunch when you see all those parenthesis, it is just the layout that is shocking, indent it well and it is no more than what you are used to.

Here’s an example

(defn fac
"Returns the factorial of n, which must be a positive integer."
[n]
(if (= n 1)
1
(* n (fac (- n 1)))
)

Is same as

(defn fac [n] (if (= n 1) 1 (* n (fac (- n 1)))))

But the first one is a lot more easier on the eyes (even brain?) than the second one. Most examples that you see look like the second one and it frightens people, don’t let that stop you take my word and go for it.

Clojure seems to be very easy to pick up, things seem very intuitive, like the other day I was wondering, how to return a default value from a map if the key is not found and there is was right there in the api.

(map key default-value)

So simple! I was easily able to extend the examples that came with the peepcode screencast. Anyways I have started on this path, let’s see where it goes.

Update 2009/12/03

– Looked at the Clojure source code, looks squeaky clean, I applied to become a member so that I can expand on the test coverage, hopefully they will accept me.

-It is (load-file “hello.clj”) and not load-file “hello.clj” , I keep forgetting that and after a few mins I realize it.

2