Here is a list of about 800 stop words made based on 4 million documents (I started with this set). This set has helped us reduce model size and increase accuracy, please note that the same list may not be applicable in your application, please review the list before using.
More interestingly here list of commonly concatenated words that I found and their corrections.
Co-workers are playing with remote-controlled helicopters, exchanging recipes for goodies, cake in the kitchen, low traffic going to work, lots of laughs in the office.
How I wish it stayed liked this for the rest of the year.
Just as we use Hibernate 2nd level cache to store data, we can also save results from a method call this is a pretty old technique and in fact functional programming languages like Haskell have this feature built in and call it with a fancy name called memoization, http://en.wikipedia.org/wiki/Memoization
Here is how it is done in spring http://springtips.blogspot.com/2007/06/caching-methods-result-using-spring-and_23.html
You can read the details in the link, but on a high level when you call a method the result is stored in the cache and the next time around the result from the cache is used, as usual you can declare how long the cache should stay active and so on in a simple ehcache.xml config file.
There is also an open source project that now lets you just decorate methods with @ Cacheable annotation and it takes care of the rest http://code.google.com/p/ehcache-spring-annotations/wiki/UsingCacheable
oh how I wish I had this at home (hint hint)
As I wait for the build I wrote up this post, has absolutely no point, just an observation. At present the code base that I work with everyday has :
1030467 lines of Java
641411 lines of Xml
224530 lines of Jsp
58950 lines of plain text
102751 lines in property files
2246 lines of groovy
1353693 lines of SQL (schema files, dml, ddl….)
90 Projects
7186 Java files
2547 SQL files
How many does yours ?
Its easy, run this find . -name ‘*.java’ | xargs wc -l | grep total | sed ‘s/total//g’
About my favorite languages, I actually have 2 favorite languages
Why Ruby is great
I have used Ruby numerous times to write scripts to fix production data, correct files, and to generate complex reports. I have used Sinatra with Google Charts to make web apps that can show load times, server status ….
Why Ruby is not so great
Why Clojure is great
I have been on a quest to learn a functional programming language for the past 3 years, I have read the Erlang book (please see the various posts I wrote about Erlang here). Erlang is a fine language but I lost interest in it after I could not find a single good library that can connect Erlang to Oracle. The problem, there are too few 3rd party libraries. The next language I looked at was Haskell, lots of libraries and seems to be good at performance on the surface, problem I see is acceptance by business, where most of the code is in Java. Then I found Clojure and fell in love with it.
Why Clojure is not great.
I have written many posts on Clojure on my blog you can see them here. In the most recent post I show you one can parse a one million record file in less than 15 seconds with clojure.
MovieLens is a research project that provides datasets of various sizes and attributes, containing movie ratings. These datasets are free to download and use for non-commercial purposes. They have done an awesome job putting this data together and a big thanks goes to them for making it available.
I wanted to exercise my Clojure skills (more like add to my tiny set of Clojure skills 🙂 ) and it just so happens that I recently came across the MovieLens project, so how about analyzing that data using Clojure ?
One of the datasets they make available is the One Million Dataset, this set consists of 3 files
I could analyze this data to answer questions such as, What age group gave the most ratings ? or What was the highest rated movie for a given time period ?
But before I could do this I wanted to denormalize the ratings file so that it also contains the user and movie information, why ? cause I don’t want to look it up when I am analyzing the data, each record should be self contained.
The outline of the program is quite simple
Take a minute to think how would you do this in java and then look at the below code. I ran it on a Dell laptop dual 2.2Ghz laptop with 4 gig of ram and care to guess how long it takes ?? scroll down for answer.
–
–
(ns com.dev.file-reader
(:use [clojure.contrib.duck-streams])
(:import [java.io BufferedReader FileReader BufferedWriter FileWriter]))
(defstruct user :id :gender :age :ccupation :zip-code)
(defstruct movie :id :title :genres)
(defn format-user [user] (str (:id user) "::" (:gender user) "::" (:age user) "::" (:ccupation user) "::" (:zip-code user)))
(defn format-movie [movie] (str (:id movie) "::" (:title movie) "::" (:genres movie)))
(defn read-user-file [fileName]
(loop [users {} fileSeq (read-lines fileName)]
(let [line (first fileSeq)]
(if (nil? line)
users
(let [tokens (.split line "::")
id (aget tokens 0)
user (struct user id (aget tokens 1) (aget tokens 2) (aget tokens 3) (aget tokens 4))]
(recur (merge users {id user}) (rest fileS)))))))
(defn read-movies-file [fileName]
(loop [movies {} fileSeq (read-lines fileName)]
(let [line (first fileSeq)]
(if (nil? line)
movies
(let [tokens (.split line "::")
id (aget tokens 0)
movie (struct movie (Integer/parseInt (aget tokens 0)) (aget tokens 1) (aget tokens 2))]
(recur (merge movies {id movie}) (rest fileS)))))))
(defn convert-ratings-file
"read the ratings file and denormalize it"
[moviesF usersF ratingsF outputF]
(let [movies (read-movies-file moviesF) users (read-user-file usersF)]
(with-open [#^BufferedReader rdr (BufferedReader. (FileReader. ratingsF) 1048576)
#^BufferedWriter wtr (BufferedWriter. (FileWriter. outputF) 1048576)]
(doseq [line (line-seq rdr)]
(let [tokens (.split line "::")
user-id (aget tokens 0)
movie-id (aget tokens 1)
user (get users user-id)
movie (get movies movie-id)
rating (aget tokens 2)
timestamp (aget tokens 3)]
(.write wtr (str (format-user user) "::" (format-movie movie) "::" rating "::" timestamp "\n")))))))
(defn doIt []
(time (convert-ratings-file
"movielens-1m/movies.dat"
"movielens-1m/users.dat"
"movielens-1m/ratings.dat"
"movielens-1m/output.dat"
)))
So ready with you guess ??
I ran the program 5 times and here is the output
"Elapsed time: 12130.035819 msecs"
"Elapsed time: 13113.92823 msecs"
"Elapsed time: 13364.234216 msecs"
"Elapsed time: 12553.478168 msecs"
"Elapsed time: 14488.706176 msecs"
On average 13.130076521799994 Seconds to read in 1 million records, for each record look up the movie and user and write it back to the disk.
Clojure puts the FUNctional back in programming.
Check out my previous post about Redis.
In this post I build a very simple example of using Redis with Clojure. I will be using a client library for Redis written in Clojure called redis-clojure. You could also use the java library, to see complete list of supported languages go to this link.
So here we go..
(defproject com.dev/try-redis "1.0.0-SNAPSHOT"
:description "simple example of using redis"
:dependencies [[org.clojure/clojure "1.1.0"]
[org.clojure/clojure-contrib "1.1.0"]
[redis-clojure "1.0.3-SNAPSHOT"]]
:dev-dependencies [[swank-clojure "1.2.1"]])
(ns com.dev.try-redis.core
(:require redis))
(defn test-redis []
(redis/with-server {:host "127.0.0.1" :port 6379 :db 0}
(do
(redis/set "foo" "bar")
(println (redis/get "foo")))))
On lines 7 and 8 we are setting key value pair and retriving the value.
If everything has gone will you should see this output.
Clojure 1.1.0 user=> (load-file "src/com/dev/try_redis/core.clj") #'com.dev.try-redis.core/test-redis user=> (com.dev.try-redis.core/test-redis) bar nil user=>
dleslie Wow, #Perforce manage to take down my entire session. Have I mentioned before how much I /love/ this software?
dleslie I hate it how that little red-X button in #Perforce /DOES NOTHING/. What’s the point of providing an interrupt function that doesn’t work?
jcsalterego @mojombo @luckiestmonkey perforce strikes again!!
jlgosse I don’t think Perforce could be any more complicated
gadorcha Fuck perforce. Kill it with fire
aidy_lewis Using #Perforce is like sticking your head in cow pooh.
art_s Wasting my life on manually merging stuff from Perforce repository with local copy. Creators of Perforce must burn in hell. Lots of hate.
Jing_Yi c’mon perforce! you fucking pig.
TimeDoctor Have I mentioned lately how much perforce
geek: For the love of all that’s decent and good, how do I remove the Perforce context menu item from the Windows shell. I hate when VCS’ do that.
Recently I have heard a lot about Redis, so I decided to try it out. But first a little intro about Redis. “Redis is a database. To be specific, Redis is a database implementing a dictionary, where every key is associated with a value.” .
Think about Redis as memcache on steroids, Redis extends the basic key/value store paradigm by letting you have values of a certain type and defines operations that are unique to each type.
For example Redis lets you associate a key with a list and then lets you do list specific operations.
lpush mylist 1 --adds 1 to mylist
lpush mylist 2
llen mylist --returns the length of mylist
By the way if you just want to try out Redis without having to download and install, then check out this link.
Supported types are Lists, Sets, Sorted Sets and Strings. Below are some interesting operations (to see the entire list go to this link).
Redis also persists data by writing to the desk asynchronously this way you can have your entire application just use Redis without the need to have a separate database.
Here is a simple example using Redis
./redis-cli set name devender
./redis-cli get name
Devender's Musings 