I've been using Amazon EC2, S3, and SQS services in various projects at work. For one project, I needed to build a little tool that integrated SQS and S3 services, and I decided to build it in Clojure. In the course of building the tool, I learned a bit about the various open source Java libraries for working with Amazon AWS, and the issues with integrating them with Clojure.
The libraries I used are:
- jets3t – S3 interface
- typica – SQS and EC2 interface
- Apache commons-httpclient and associated dependencies
- postgres jdbc driver
Having been away from Java development for some time now, I'd forgotten about the convoluted mess of Classpaths and Class loaders. I had a "crash" refresher course in the last week. And, it's all still a bloody mess. I did come across one-jar, which made it pretty easy to package up the various jar dependencies into a single deliverable jar file.
In the end, the tool worked out well and served its purpose. Plus, it was rewarding to be able to write code like this:
(defun process-loop [:key
aws-access-key
aws-secret-key
geo-db-path
[message-count 5]
[sleep 5000]
sqs-queue]
(with-geo-service (geo-get-service :path geo-db-path)
(with-sqs-queue (sqs-get-queue aws-access-key aws-secret-key sqs-queue)
(with-connection *db*
(let [process (fn [msgs]
(if (empty? msgs)
(Thread/sleep sleep)
(try
(transaction
(doseq [msg msgs] (process-message msg)))
(doseq [msg msgs] (sqs-delete-message msg))
(catch java.sql.BatchUpdateException ex
(println "* process error:"
(.getNextException ex))))))]
(while true
(try
(process (sqs-receive-messages message-count))
(catch Exception ex
(println "* error:" ex)))))))))
and know it's running on the JVM.
I've committed some of the code for the Amazon integration into the cynojure git repository, in case anyone else finds it useful.
Harmony.