I used the points accumulated into my Vodafone account and got a new Blackberry with 0€, which is nice. I was using a blackberry 8700 for a long time, so will be nice having camera and wifi on it.

I have an iPod Touch which I use a lot, so guess browsing will still use the iPod, but now I will have camera to upload pics to facebook, etc… as well as video. The device is GPRS/EDGE Quad band which is nice for travelling.

Can sync with my Exchange server and have same things on my iPod and Blackberry, email, contacts, etc… while on Wifi. I do not expect to have a BES as I used to have with my blackberry due to cost and I can get along downloading stuff while on Wifi.

A nice thing is to have the newer blackberry OS and play around with blackberry apps. Besides, having keyboard is nice for writing my long notes as I used to do and sync them.

Why don’t get an iPhone?

Well, first is cost, I would go back to a costly solution like I had with the BES. Second, will not move to Movistar no matter what. And thrird, is nice having two devices, one long battery one and the other fun and short battery. Sometimes happens to me that iPod is out of battery, but still my BB still can use it in Metro for example to have maps, play, etc…

Tweet This Post Plurk This Post Digg This Post Ping This Post

Last weekend I worked on the design of the new modules for OctopusDbm and looks like nice features are coming in soon. The first release was built, 0.6 and soon I will start documenting the current modules and design.

Community

Any open source project needs a community or group of people that like the technology and contribute with ideas, code, testing, analysis, etc… It will be a nice experience for me to learn how to do this. So the following weeks I will start promoting the project and trying to attract lovers for this project. Spread the word if you read this, this tech can be pretty cool.

Dbm

The client/server implementation works in such a way that the code change for applications already running BerkeleyDB software local is minimal. Almost all API methods are implemented in remote server.

Curently you may have:

from bsddb3 import db
Dbm = db.DB()
Dbm.open(sPath, db.DB_BTREE, db.DB_CREATE)
Dbm.put('Name', 'Jorge')
Dbm.close()

And with OctopusDbm:

from octopusdbm import Dbm
dbm = Dbm(hostName, user, password)
dbm.open(sPath, db.DB_BTREE, db.DB_CREATE)
dbm.put('Name', 'Jorge')
dbm.close()

So you basicly can open, put, create cursor, get, get table information, etc… same as you do in local. Keep in mind that for every database methods there is a call to the server. So server connections for open, put and close. If you need better performance we have direct operations where open and close are integrated, like getDirect(), getLIstDirect(), as well as bulk operations like getList() and putList() with tables opened.

No Db Sessions

One of the design points of OctopusDbm is that you don’t need to open connections to database or create sessions before processing operations in database. The idea is that getting and putting data is as simple as getting an HTML page, so simply a request is performed and you get result. This way is easier to connect to 10 or 20 cloud images without pool of connection, etc… Imagine having a pool of 3 connections to 100 cloud images, that would be 300 database connection to manage for every node needing database connectivity to the cloud.

This is of course open to discussion, so place your ideas, etc…

DbmCloud

We introduce this object for access to the cloud. The distribution has been designed so far to e horizontal. That is, table 1 goes to cloud image 1, table 2 to cloud image2, …., table 11 to cloud image 1 when having a cloud with 10 images or virtual machines. This object also manages partitions in horizontal way and while you have an integer primary key or index. So tables being in the cloud must have integers if you want to partition them. If they have string indexes they will not be partitioned so far. I think this is ok for first versions. You can allways pass this DbmCloud, implement your own, participate in the project with your own dessign of cloud distributions and partitions, etc…

It has been decided so far to have an XML file with all schemas, tables, defining number of partitions, number of cloud images organized by domain and application name. For very big systems, they could have different domains and application inside each one with a cloud design. Small apps would have 1 domain and 1 app.

One cool aspect of DbmCloud is that any database object, that is, data table, index file, foreign key will be distributed and queries running on them executed in different cloud images therefore boosting performance, even more when smart partitioning in place.

Models

In order for OctopusDbm be an alternative to SQL databases like MySQL we need a model design that will:

• Allow us to have tabulated data in columns
• Design to make it easy to implement persistent subsystems like Hibernate, JPA, etc…

We thought to do this inside Python similar to Django models, where you define class attributes with classes relative to types and using dictionary class attributes:

oClient = Client(Name='Jorge', Place='Madrid')
oClient.save()
 
oClient = Client.get(Name='Jorge')
oClient.Place = 'Madrid'
oClient.save()

First one creates an object in database system and writes to database in a lazy mode. Second one updates the client object in database. I think the models would support cloud distribution and partitions. You define in the class the attribute types, index names, physical names, etc,….

The primary key in tables can be an id or other value. There will be a type IdGenerator that will generate auto ids for rows, but you could define you own primary key values. When support for cloud and partitions we need an integer primary key for partitions.

First versions will probably lack foreign keys, but later on we can implement foreign keys with model integrity. I thought of Links between tables besides ForeignKey types to execute soft links for example to get literal of an id in a parametric table.

Tweet This Post Plurk This Post Digg This Post Ping This Post

Nice article about taking notes software here:

If You Didn’t Write It Down, It Didn’t Happen — Take Notes

I used OneNote from Microsoft and was nice the visual aspect of the tool, but stopped using it, I guess becaouse I started to take notes on my blackberry, sync with Exchange, etc…

My opinion is that the networking of these tools is very important. I signed up for Evernote but never catched me up enough to feel great about the tool.

I would add Twitter as note taking, many people use Twitter to do that, write down things so he and others can take a look at it later on. And could ot be simpler note taker than Twitter, right?

Tweet This Post Plurk This Post Digg This Post Ping This Post

I tested new Oracle Java edition for Berkeley DB and for my tests the word that best describes the product is FAIL. The conslussion more or less is that Oracle screwed Berkeley software. They have no HASH, only BTREES, many functionality of BerkeleyDB has been removed, etc… They say is BerkeleyDB with Java flavour, they have a weird sense of java humor then.

I was expecting factor of 5X as many blogs suggested that on real applications we have this improvement. And since BeerkeleyDB for me has a lot of memory use and processing as well I was expecting at least 2X factor. I got a 2X factor but on the contrary.

Results

I did test for creating a table of 1 million rows with 128MB memory Cache.

So the refactoring of OctopusDbm, if any, will go through C++ inside Linux. I downloaded the Linux Tools for Eclipse and will start playing with those when I have some time.

It seems that Python with C modules work great.

[Update Feb 22 2010]

The tests were performed 5 times and get the average, plus:

• BerkeleyDB.4.7
• bsddb3-4.7.2 (Python API)
• Linux SuSE 10.1 Pentium 4 Hyperthreads
• Kernel 2.6.16
• Sun Java 5

And here is the code used:

/**
 * Testing for BerkeleyDB
 */
package test;
 
import java.io.File;
import java.util.Date;
 
import com.sleepycat.je.*;
 
/**
 * @author Jorge Alegre
 *
 */
public class BerkeleyDB {
 
	private final static String HOME = "/index/test/";
	private final static long CACHE_SIZE = 128000000;
	private final static Integer MAX = 50000;
 
	/**
	 * @param args
	 */
	public static void main(String[] args) {
		// TODO Auto-generated method stub
 
		EnvironmentConfig envConfig = new EnvironmentConfig();
		envConfig.setAllowCreate(true);
		envConfig.setCacheSize(CACHE_SIZE);
		Environment env = new Environment(new File(HOME), envConfig);
		System.out.println("Data: " + envConfig.getCachePercent() + " " + envConfig.getCacheSize());
		DatabaseConfig dbConfig = new DatabaseConfig();
		dbConfig.setAllowCreate(true);
		long t_1 = new Date().getTime();
		System.out.println(t_1);
		Database db = env.openDatabase(null, "testing", dbConfig);
		DatabaseEntry key = new DatabaseEntry();
		DatabaseEntry data = new DatabaseEntry();
		String sData = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx";
		data.setData(sData.getBytes());
		for (Integer i= new Integer(0); i

Tweet This Post Plurk This Post Digg This Post Ping This Post

Last night I run the Spring IDE for Java. Apart from the fact that Eclipse in my laptops is slow due to low memory, the Spring IDE seems to be heavier than Eclipse. I am used to work with python and Eclipse using pydev, and start-up times are poor, but Spring seems even worse, so like 10 minutes waiting for things to load, etc…

The welcome page looks great, with information from Spring web site, feeds, and download center with cool software like SVN repository, sync of development task with MS Outlook, Trac integration, Grails, Groove, etc… When trying to install those applications got an error, so man, bad luck to me.

Then created a Java project Buscaplus and tested BerkeleyDB. The experience of IDE is pretty nice, getting nice suggestions with the autocomplete.

I still can’t seem to work out the Subversion repository from Google code, will try this weekend. the default repository is CVS, and since the easy-to-install functionality did not work, will have to go the long way.

I run tests for BerkeleyDB and I was not impressed by performance. Aprt from this, Oracle has changed the implementation for Java from the original from Berkeley, so you only have BTREEs, can’t create tables without environments, etc… This was a total blowout, I can’t use it for OctopusDbm, I prefer an original implementation of BerkeleyDB and Tokyo instead of a Java flavour. As part of this, Java implementation does not physically writes data in the xxxx.db file you determine, but creates partitions of about 10MB each. I don’t like this, prefer the original way. If I choose to have a file 500MB with a cache of 128MB then the software should do what I need. So a C++ will have to be worked out for OctopusDbm.

Going back to Spring, I missed an integrated application server coming with the install of the suite. I thought that would come with the installation, so will have to install tmServer based on Tomcat someway.

As a conclusion, Spring IDE is ok, but had problems with me. Many things are integrated like Aspect projects and Spring views and perpectives are cool, but gee, some things did not work and I don’t like that.

[Update Feb 22]

I installed again Spring and everything went perfect. For some reason first install had problems with Equinox. I now see installed the dmServer (Development) and tcServer (Tomcat) as well as the download center works ok.

Tweet This Post Plurk This Post Digg This Post Ping This Post

Last night I installed Spring Tool Suite, the Spring IDE and started to digg into their world.

It seems Java is moving in right direction, making software easier to maintain and speeding up development. This is particulary important for startups, time to prototype and time to beta is crucial many times.

Spring alone simplifies things a lot. But when you add Grails (Like Ruby on Rails for Java) and Groove (New simple syntax), suddenly you are in middle of something weird, Java but not Java. No wonder why this stuff has been sky-rocketing.

You like POJO’s? I found that the proyect I work for eliminated them and instead want the old EJB syntax. I think Corps have been used to complex things out, when you simplify things some of them don’t like it.

But anyway, sometimes innovations like RubyOnRails and Django don’t have enough traction to move the market but competition starts to copy those innovation into a market mover like Java.

And Java is redifining itself, which I think is good for startups. If your startup suddenly needs 20 people in 6 months, good luck finding Python or Ruby people in the market. But if those developers are Java, lot’s of them in the market. Another reason why this is important.

Tweet This Post Plurk This Post Digg This Post Ping This Post

Today I was doing testing on different Amazon EC2 images with bulk writing activities usually performed in my startup Buscaplus, an Internet search engine framework. Currently I have a set of 4 servers with SATA disks and I am planning to move to Amazon.

We use Berkeley DB as index database engine. It is pretty fast, specially  if you define correctly the memory cache, etc… In Buscaplus we need to write huge amounts of data to disk and bottlenecks are often found due to the high database requirements for a search engine. So this is crucial if we ever move to amazon, speed of writing stuff to disk. A deployment and cloud design for many instances has not been accomplished but with today´s tests seems clear that Amazon EC2 is an option for Buscaplus.

Tests

Berkeley DB writes data in key-> value sets. You can select BTREE as well as other engines. We use BTREE and a cache spool of 128MB for all tests. Also, we write 100 Bytes for each row of data. The keys are simply a counter with zeros on right, like ‘0000000345′.

index-1 shows one of the current servers. I would conclude that the “medium” instance is a great option. At only $0.20 / hour has great performance, better than current infrastructure.

I also found that when dealing with a lot of data, small instance of course are a “no-no”, but also higher instances with local disks. I noticed that when dealing with high I/O even big instances may do bad if load at that time is high. I found that this is not the case when having EBS. With high I/O and EBS I got great results all the time. So I would go for sure with EBS.

The 20 million rows tests were unstable even with a $0.80 High CPU Extra Large instance. This ended up in a DB table of more than 3GB.

EBS

So EBS does a great job at performance and stability (no load bottlenecks found).

Also was found that disks mounted in / had better performance than those in /mnt and EBS volumes.

Conclusion

I think testing in the cloud could lead to wrong conclusions. The testing data will vary as the load in the server hosting your instance increases or decreases so it is a good option to repeat tests in different time period, like morning and night or something similar.

In my particular case where I need lots of writing and reading I would go with medium instances for the database nodes and probably medium instances for processing nodes. I would stay away with small instances since the I/O it is very bad. Would get small instances only for testing purposes, development servers or early stage beta when load is not high.

I prefer to pay for 4 medium instances ($0.80) than for one $0.80 instance.

Tweet This Post Plurk This Post Digg This Post Ping This Post

A couple of weeks ago I went to a couple of events that have influenced me a lot, thinking days after those events the advice and knowledge shared. One of them was the founder of Panda Security, Mikel Urizarbarrena, and the other the founder of Dinamic Multimedia, makers of PC Futbol by Jesus Alonso.

The case of Panda Security shows that it is possible to start and develop a technology business in Spain without venture capital and do that even internationally. This guy is really unique, reinvested profits to make the business grow, without government help or investors, just by listening to the client needs and be able to manauver and be productive compiting with others. After some years developing the business, he has left the management part of the company, run by a experienced manager and some partners from a venture. They have plans to grow in the USA, which leads to my second conclusion: No matter what, for a tech company the US connection is a need, it is needed to be strong in that market, have good partners, etc… They implied that if they grow better than expected could go to stock exchange in the US. My conslussion is that I must design and define carefully the US connection for my tech startup so that grows in that markets and has great aliances, but having attachments in Europe.

The second event was great too, came a guy that revolutioned the PC gaming industry in Spain back in the 80? 90? He was a very sales-oriented guy with great communication skills, so the advices to entrepeneurs were really interresting. He said that when the bank calls you asking how the business is going and you know that if not running good, the bank will get your house and you will have to sleep under a bridge, that you somehow get so much motivated that if you are good, something great will come out, like the FX Interactive game they developed. So this guy also a 4×4 of business activity. Since I am in that category, needing commercial activity desperately I understand that pretty good. I will not loose my house fortunately, but will have troubles.

So no wonder why entrepeneurs in Spain are lacking, it is like jumping with no parachute: If things go wrong and only way to get money is to offer your house as guarantee, you not only loose your house, you will have to pay for the rest of your life the difference between the price sold y bank and the mortgage, plus you will owe any workers fees for the rest of your life too. It is not like the US whene is game over, starting a new game. Here is game over, and you are done! That is why people here when you fail you fail for the rest of eternity. In the US is like “great, you failed”, let’s start a new thing now that you learnt.

So the final conslussion is that entrepeneurship si so damn hard that we must be fucking crazy, right? We do it cause we love innovation and starting things up, but can be pretty dangerous, expecially in Spain. I also like living in Spain but I agree that somethings are harder here, which leads me to the conclussion that harder=more motivating

Tweet This Post Plurk This Post Digg This Post Ping This Post

Today is a celebration for my name, ‘San Jorge’, so will be a happy day, I had ‘Churros con chocolate’ for breakfast, so the day started great already.

I read some posts in twitter about Yahoo and the MS deal. For me Yahoo (YHOO) is already history. It belongs more to wikipedia than today’s Internet. They are out of everything, either if they take the. Google deals or the MS deal. If they take the G deals, it will be a suicide in the long term. If they take MS it will be a grave illness called fusion, and 2 recovering. If was investor, I would sell YHOO at a right price and buy GOOG. Money is not romantic, it is just money.

I am going to work in metro today for first time. Let’s see how long it takes to get there.

I installed Outlook2007. For me it is not great, it seems goes slower than Outlook2003, but has better presentation and cool feature, so I guess I will keep it. I got an error while installing, was trying to read a .CAB file from DVD when I installed it from a Directory. Probably will save it into DC/DVD and reinstall it. Maybe the perf is due to the intsllattion problem.

We are having primaries in the political party PP here in Spain, but in the spanish PP way. This means that a candidadte says will present in party election and the leader says will kick her out from party. Great democratic culture. A message to those politicians: ‘Learn from the US, party democracy is good, don’t be afraid’.

That is all from today

Tweet This Post Plurk This Post Digg This Post Ping This Post

Last week I got some vacation from last year, 15 days. It was great, I was hilarious about it.

Last year I had some problems with my legs. Due to some reason, they are not very strong, could not stand too long, walk for more than 5 min, I even could not stand while having a shower.

Last January something happened (which I yet do not understand), and improved the thing a lot, being very happy about it. Unfortunately, the improvement did not continue, and right now I can do almost normal life, got back to the showers but I cant do other things like travelling, take the car for long journeys, etc… Which actually sucks…

Although I tend to be very possitive about it, the real thing is that I dont know what else is to do to feel better. I started with a new therapist (fisio), made accupunture,  changed my food habits (more mediterranean food style), go to swimm 4 days a week….

So I will continue with the search of well being . Tomorrow I will take Reiki sesion, which I guess could help. And probably soon I will start Yoga. Could be that my nervous system is sending signals to muscles, and they dont relax as they should. And when I start walking, they soon get saturated. This is soooooo weird. I am starting to be pretty nuts about this stuff.

I am starting to consider how should I compromise extreme work loads and my health. I love to pursuit impossible things (right, I am a sagitarious), but dont want my health to suffer. I guess some of these disorders may be well caused by my fight with my search engine system, the problems in the market to compete with Google, etc… When I was in University in Kansas, a professor came at a workshop and advised us: “Never, ever, ever, ever, ever give up”. I thought that to be a silly phrase, but later on I assumed this as a way of life. So basicly, I will never give up in pursuing the right for search technology engineers to survive even being Google and all the big empires.

Working in a company and allocating 13-16 hours a week after job doing programming for the search proyect has been probably too much, so I will break a little, and focus more on my health, now that everything is under way.

If would be easy (or hard, depends how you look at it), to give up and say: “man, I cant take it anymore”. I would relax a lot and spend more time in being happy, my weekends for me, cool. But after some time, my I would have felt that I gave up on my dream, and could not take that. But I will try to take it easy, so that all these google mania dont affect me. Now I have to take care of my health. Probably will seek help in programming (paying of course, and things like that).

Almost everybody I meet in Spain thinks my venture is a lost battle (even if they say it or not), but 90% of those people probably said the same thing when Google got the $20 million: Yahoo is a great brand, there is not place for a new search service… So we end up at the same place: “Never give up”. That is the 101 of internet entrepenours.

I think that I need to work on my mind, so that mind and body are balanced. Yoga could be great for this, so my inside energy is balanced.

Anyway, all this thing is a new journey for me, I hope to learn if anything is to be learnt from this small adventure.

So for these days I will be a tourist in Madrid, go to musseums, paseo del prado, riding on the no-top bus, sleep, go to nice restaurants, etc…

Tweet This Post Plurk This Post Digg This Post Ping This Post