Category: Tech

Tech

Smart Databases : RethinkDB

Posted on by devender

I started playing with RethinkDB and it is very interesting and definitely worth a look.

Lets start with a typical example in applications for example the client table. The client table can get modified via UI or SalesForce or by hand or any number of ways. Hence any app cannot just read this table once and keep it in cache, we have to constantly pool to get changes.

This is very frustrating, cause this table rarely changes but if it does change we need to know about the change immediately, so far we have relied on polling or the system that makes the change inform other systems so that they can blow there local cache.

But with RethinkDB it changes all that.

RethinkDB is like a smart database, your client can listen for table changes and whenever that particular table changes the database itself will notify any clients that the table has changed and on top of it will tell you the old and new values.

This is almost revolutionary, (yes I know you can do this in mysql by installing the JDBC driver and a trigger that listens on the table and the trigger can inform other apps but it is way too complicated and good luck getting your ops to install it), but RethinkDB does it for you OUT OF THE BOX. And they now have an awesome JDK driver.

Just awesome sauce.

3
General, Tech

Neo4j: Running Embedded Server with WebConsole

Posted on by devender

Took me couple of hours to figure this out so blogging it, hopefully it helps someone else.

If you are running Neo4j in embedded mode, you can still get the web console, data browser and other goodies, they do mention this in the manual but what they don’t mention is that you will need 2 extra jars to do this neo4j-server.jar and neo4j-server-static-web.jar and these are not available on neo’s repo, so you will have to clone their source from git and build it locally.

Add them to your pom.xml

<dependency>
  <groupId>org.neo4j.app</groupId>
  <artifactId>neo4j-server</artifactId>
  <version>1.5.M01</version>
</dependency>
<dependency>
 <groupId>org.neo4j.app</groupId>
 <artifactId>neo4j-server</artifactId>
 <version>1.5.M01</version>
 <classifier>static-web</classifier>
</dependency>

Notice the “classifier” in the above code. Below is the code for how you would start it.

EmbeddedGraphDatabase db = new EmbeddedGraphDatabase(<path>);
bootstrapper = new WrappingNeoServerBootstrapper(db);
bootstrapper.start();

UPDATE
Once you get the web console you will be able to run Cypher queries but not Gremlin, to be able to run Gremlin queries too include it into your classpath. 

<dependency>
        <groupId>com.tinkerpop</groupId>
	<artifactId>gremlin</artifactId>
	<version>1.3</version>
	<type>jar</type>
	<exclusions>
		<!-- Sail support not needed -->
		<exclusion>
			<groupId>com.tinkerpop.blueprints</groupId>
			<artifactId>blueprints-sail-graph</artifactId>
		</exclusion>
		<!-- Maven support in groovy not needed -->
		<exclusion>
			<groupId>org.codehaus.groovy.maven</groupId>
			<artifactId>gmaven-plugin</artifactId>
		</exclusion>
		<!-- "readline" not needed - we only expose gremlin through webadmin -->
		<exclusion>
			<groupId>jline</groupId>
			<artifactId>jline</artifactId>
		</exclusion>
	</exclusions>
</dependency>
1
Tech

Maven Integration Tests

Posted on by devender

Ever forget to add @Ignore to your integration test and have the rest of the team complain or create a different project just to hold the integration tests, well no more.

With the maven failsafe plugin, you no longer need to ignore your integration tests. This plugin will pick up any tests that have *IT*.java in them and run it for you.

Continue to run you regular tests with ‘mvn clean install’, if you want to run your integration test run ‘mvn failsafe:integration-test  failsafe:verify’

Don’t forget to remove the @Ignore and rename your tests to *IT.java.

0
General, Tech

Clojure, JDBC and using Dates.

Posted on by devender

I did not find many examples of using dates with Clojure/JDBC, after some initial struggle I got an example working. So here it is, hopefully it helps someone. By the way this page is a good source if you are interested in Clojure/JDBC.

Source File

(ns clojure-bi.simple
  (use clojure.contrib.sql))

(defn connection-props [connect-string user password]
  {:classname "oracle.jdbc.driver.OracleDriver"
	 :subprotocol "oracle:thin"
	 :subname (str "@" connect-string)
	 :user user
	 :password password })

(defn dev-db-props [user password]
  (connection-props "devdb.db.cj.com:1521:devdb" user password))

(defn make-sql-date [year month day]
  (java.sql.Date. 
   (.getTimeInMillis 
    (java.util.GregorianCalendar. year month day))))

(defn get-company-name [db id]
  (with-connection db
    (with-query-results rs ["select organization from company where id = ? " id] 
                  (first rs))))

(defn find-by-date [db date]
  (with-connection db
    (with-query-results rs ["select organization from company where trunc(Date_) = ? " date]
      (first rs))))

Test file

(ns clojure-bi.simple-test
  (:use [clojure-bi.simple] :reload-all)
  (:use [clojure.test])
  (import (java.util Date)))

(def dev-db (dev-db-props "devender" "test"))

(def expected-company {:organization "Client Services"})

(deftest get-company-name-test
  (is (= (get-company-name dev-db 2) expected-company)))

(def date-to-use (make-sql-date 1999 6 15))

(deftest find-by-date-test
  (is (= (find-by-date dev-db date-to-use) expected-company)))

The interesting parts are the find-by-date function in the source and the test function find-by-date-test, you have to pass in a java.util.sql Date.

1
General, random thoughts, Tech

IMHO

Posted on by devender

TWIKI sucks: plain and simple, the end. I have had to endure it for the past 17 months and cannot find a damn thing one it plus editing sucks. A colleague finally decided to install a search/index engine on a separate box. I guess I got spoiled using Confluence in my last job, it was the best.

Respect my authoritah: It is imperative to have a decision making authority in every meeting (and not a quiet one either, but someone who will put his foot down) or the discussions never stop. This goes double for email discussions, sometimes I wonder how people can get anything done if they spend all their time in typing up long emails (and yes I have written my fair share of long emails too). Anyway here is a great quote from the book ‘The Mythical Man Month’ – “We finally decided to implement the language unchanged and unimproved for the debates about the language would have taken all our effort“.

Perfect is the enemy of done: Along the same lines as above, I have seen many spend all their time and effort revising and redoing to achieve the perfect Object Oriented Design and at the end of the day have nothing to show for. Another quote, unfortunately don’t know where I read it, “A painting is perfect when you cannot take anything away from it” or in other words know when to stop. Although one should strive for good design, that should not be your end goal. The end goal should be ‘getting things done’.

Importance of having a testing team: Even if you have 100% coverage (I think that it is a myth). You need someone whose whole job is to break your system, there is ton of literature on the importance of an independent testing team but unfortunately I keep seeing many places where it is done very less or totally absent. Also I think it is equally important to load test the application before deploying it.

3
General, random thoughts, Tech

RE: Software Metrics Don’t Kill Projects, Moronic Managers Kill Projects

Posted on by devender

Great article Software Metrics Don’t Kill Projects, Moronic Managers Kill Projects, couldn’t agree less. I had written about working for good managers a while ago, thinking back about it and in general about the rate of successful Software projects I can’t help but think, Does the software industry in general have a high rate of Moronic Managers ? how come the software industry has so many failed projects ?

I think part of the problem lies in the fact that most managers have a technical background, they are great programmers and their success was rewarded with a promotion. Wish there were some online program or cash course that these new budding managers could take. I know there are a lot of books that can help too. I think it should become mandatory that one has to either read or get certified before he becomes a manager.

We have a saying in India, ‘For an intelligent person, you just need to show him the direction”, he will do the rest, I think programmers in general are very intelligent they just need the right direction 🙂

1
Tech

Why do we need an SCM ?

Posted on by devender

So I have so far worked in four different IT shops of various sizes and noticed in all of these shops, the person who ends up merging code is a Senior Developer/Tech Lead and in most cases the SCM and the senior developer are the same. This senior developer merges all the changes gets with the developers if he has questions, builds and check in the code (Even if there is another designated SCM).

This reminds me of a place I used to work, where the most brilliant Senior Developer who should have been doing other things like designing and coding was spending most of his time merging code cause the SCM was too dumb to read the SVN book.

I also notice that the senior developer is the guy who sets up Cruise Control/Hudson and then hands it over to the SCM (i.e if there is one).

So here is my question “Why do we need an SCM” position when the senior developer is doing all the work ?

Now if you are using Clear Case I totally understand and feel sorry for you.

1
General, ruby, Tech

Use Ruby to Prefix all lines /Postfix all lines/Remove empty lines/Replace strings in all lines of a file

Posted on by devender

For some reason or the other I had to do bulk operations on files, like prefix/postfix all lines or remove empty lines
, yeah I could do it with some unix command, but I am already using ruby to massage the data just wanted to extand that.So here
is what I did wrote a ruby script to do it now I can do stuff like

prefix ( ‘c:\tmp\zip.txt’,’some-prefix’ )

The script, hmmm I wonder if I could submit this to ruby people.

require 'fileutils'

module DevFileutils
include FileUtils::Verbose

# Opens a file and adds prefix to each line
def prefix file_name, prefix
temp = File.new(file_name+".tmp", 'a+')
IO.foreach(file_name) do |line|
temp.puts prefix + line
end
temp.close
mv file_name+'.tmp', file_name
end

# Opens a file and adds postfix to each line
def postfix file_name, postfix
temp = File.new(file_name+".tmp", 'a+')
IO.foreach(file_name) do |line|
temp.puts line.chomp + postfix
end
temp.close
mv file_name+'.tmp', file_name
end

#removes all empty lines from a file
def remove_empty_lines file_name
temp = File.new(file_name+".tmp", 'a+')
IO.foreach(file_name) do |line|
temp.puts line if !line.chomp.empty?
end
temp.close
mv file_name+'.tmp', file_name
end

def replace file_name,string,with_string
temp = File.new(file_name+".tmp", 'a+')
IO.foreach(file_name) do |line|
temp.puts line.gsub(string,with_string)
end
temp.close
mv file_name+'.tmp', file_name
end
end

7
Tech

Bank Of America Login Script

Posted on by devender

def login
@id = ‘USERNAME’
@pwd = ‘PASSWORD’
puts “logging in ..”
ie = Watir::IE::start(‘http://www.bankofamerica.com/index.cfm&#8217;)
ie.set_fast_speed
puts “entering online id”
ie.text_field(:id, ‘id’).set(@id)
window = ie.ie.Document.parentWindow
window.execScript(“doPassmarkSignIn()”)
puts “Enter passcode”
ie.wait
ie.text_field(:index ,1).set(@pwd)
window = ie.ie.Document.parentWindow
window.execScript(“verifyImageForm_0_submit()”)
puts “done.”
end

8