The Slog

I’ve spent a few hours over the last few days working on kdb+ IPC. Essentially it is very similar to the IPC in kdb.

IPC stands for inter-process communication. Currently kdb+ supports IPC via tcp/ip.

A kdb+ instance can be a server, client or both.

A server simply listens for connections on a port. kdb+ can currently only listen on a single port for incoming connections. This can be done through either the command line,

q -p 2001
or the q command

\p 2001
To stop listening, you can ask it to listen on port zero, i.e.

\p 0

A client connects to a server. E.g. if the server is on the same machine

h: hopen`:localhost:2001:username:password
which stores the handle of the connection in the variable h

If you want to invoke a function on the server, first make sure the function exists on the server, e.g. at the server console type

f:{x+y}
to define a function that will add the implicit x and y parameters. And from the client side, after opening the connection, you can invoke the function as

h(`f;2;3)

which should give you the answer 5. Note this call blocks the client until the function f returns, or the server instance dies/exits.

If you want to invoke a function on the server without waiting for any kind of confirmation you need only send an async message, e.g. having defined a new function on the server as

f:{}

it can be invoked as

(neg h)(`f;())

Note that to send an async message, you need only negate the handle. kdb users will also notice the change in list notation,
i.e. (`f;()) instead of (`f;).

(`f;) returns a projection of a list rather than a list of 2 elements with the second element as null.

Simon’s just reminded me of a pretty neat IPC feature, and that is you can invoke lambdas (unnamed functions) in this manner too, e.g.

h ({x+y};2;3)

or

f:{x+y}
h (f;2;3)

So that’s pretty simple. You can close a connection with

hclose h
NB: hclose and exit flush pending messages.

If you want to send a bunch of async messages, and then wait for them to complete processing, you can chase them with

h”"

which will cause the client block until the server sends a null reply to that message.

At the server side, you may be interested to get the users socket handle, username, ip address etc

so at the server side, rewrite the function f as

f:{-2 string .z.w;}

.z.w - socket handle
.z.u - username
.z.a - ip address

And then if you invoke these from a client you will see the information printed on the server console.

Hooking up java, c# and c/c++ clients to kdb+ is child’s play. You may as well forget jdbc - you’ll get much better performance from the direct java sockets driver (c.java) and is much simpler to use. e.g. to get a list of tables present in your database you need only do this

c c= new c(”localhost”, 2001);
String[] tables= (String[]) c.(”tables`.”);
c.close();
You’ll probably want to wrap it with try/catch/finally etc, but I think you get the message.

kdb+ Stored Procedures are really just q functions, like the f function that we wrote above. An example query may be

myproc:{[symbols;start;end] select price from TRADE where sym in symbols, time within start end}
this can be invoked from java as

c c= new c(”localhost”, 2001);
c.Flip table= (c.Flip) c.(”mpyroc[`VOD.L;12:00:00;12:10:00]”);
// read the data from the flip struct…
c.close();
I’ve just rewritten the c.java to use java nio - (java’s new io model. It keeps as much data outside of the vm as possible, so can be faster for some things, e.g. streaming data directly from disk down a socket). I created a java server that looks like a kdb server, so a kdb instance can connect to it, and invoke some java functions (the receive msg event handler uses reflection to locate the method to invoke). This is another way of extending kdb+; e.g. this framework could be used to interface to other systems, databases. Essentially a function like this in java

public Integer add( Integer a, Integer b)
{
return new Integer(a.intValue() + b.intValue());
}
could then be called via IPC as

h(`add;1;2)

without adding anymore logic to the receive msg event handler.

You’ll not want to use this to add integers, but it shows the essence of how simple kdb+ IPC is, and how this framework exposes systems that integrate well with java can also be exposed to kdb+ instances.

That’s all for now.