Wow, seriously, hats off to Google on this one…  This should make threaded apps programming easier in the future for me.  They have created a language called “Go”, which in my opinion, looks like if Python and C++ had a child out of wedlock.  I don’t think I’ll be able to stop gushing about Go for a while.  I believe I have found my new prototyping language, however, I don’t see a reason why this couldn’t run production code…

Check out this snippet of Go code:

file: threads.go

package main

import "fmt"

import "syscall"

const THREADS = 2;

func threadtest(a int) {

fmt.Printf("Thread %d started.\n", a);

syscall.Sleep(10);

for true {

for i := 0; i < 10; i++ {

fmt.Printf("Thread %d -> %d \n", a, i);

}

}

}

func waitforever(debug int) {

for true {

if debug == 1 {

fmt.Printf(".");

} else {

fmt.Printf("");

}

}

}

func main() {

fmt.Printf("testing threads:\n");

for i := 0; i < THREADS; i++ {

go threadtest(i);

}

waitforever(0);

}

Output:

$ ./threads

testing threads:

Thread 0 started.

Thread 1 started.

Thread 0 -> 0 

Thread 1 -> 0 

Thread 0 -> 1 

Thread 1 -> 1 

Thread 0 -> 2 

Thread 1 -> 2 

Thread 0 -> 3 

Thread 1 -> 3 

Thread 0 -> 4 

Thread 1 -> 4 

Thread 0 -> 5 

Thread 1 -> 5 

Thread 0 -> 6 

Thread 1 -> 6 

Thread 0 -> 7 

Thread 1 -> 7 

Thread 0 -> 8 

Thread 1 -> 8 

Thread 0 -> 9 

Thread 1 -> 9 

Today  I realized the strengths and speed of Lucene, but I also realized something else: Zend_Search_Lucene is painfully slow compared to Java’s implementation of it, and it requires a ton of memory to perform Lucene searches.  The ease of development, in this case, is not worth the hassle (and the memory leaks).

run time

The great alternative, albeit a pain in the ass to implement (due to debugging and not playing friendly with IDEs), is the PHP/Java Bridge method of doing a Lucene search.  I was able to perform a sample search against a Lucene index with Zend’s implemention in an average of 0.30 milliseconds (varying by 0.1 milliseconds) with a required 50mb of memory allotted to the PHP script on an index containing 29,000 documents.  I was able to swap out the Zend-specific objects and replace them with PHP/Java Bridge SOAP/Java methods.  I really didn’t think I had much to gain by doing this other than cutting down on my memory usage.  I thought just by the pure fact it had to make a socket connection and then make calls to the java Lucene jar file, I was going to lose time… I was totally wrong on this one.  To run the same query against the vanilla tomcat server running in local mode through the PHP/Java Bridge took a consistent 0.01 milliseconds with almost no variance.

server load

How do they compare side by side with a load test?  This one shocked me too – the Zend_Lucene_Search implementation would theoretically max out a 4gb server memory with 80 concurrent connections.  By today’s standard, a load of 80 concurrent users shouldn’t max out the server’s memory with an index size of 29,000 documents (with no extraneous binary storage) consuming ~15mb on the disk.  That is totally unacceptable.  The PHP/Java Bridge implementation survived well over a load of 1000 concurrent connections without maxing out the memory or the processor.  The only problem I ran into was “too many open files” which a “ulimit” modification (with a tomcat restart) solved.   However, I do not expect there to be thousands of connections to just this one server opened at the same time – I just wanted to see how much it could handle … and it turned out to be quite a lot!

overall

Hello everyone, in the coming weeks you’ll see some of my newest projects come to life, stay tuned for more!