Wednesday, February 29, 2012

Doing high speed appends with Sqlite3

While maintaining Qson nested format. Simple, the default init_table context will created pre-compiled statement to the table named. The process converts a Json package automatically to Qson in the parser. But all that is is append table, or append and update forward pointer. The engine just opens a table, pulls out the lead triplet whose bounded object is the entire table. But it is a bit difficult to do inserts, the whole or part of the Qson needs to unpack for insert. The preferred is a fast rewrite of the table. But streaming onto append, for regular word lists, regular tweets, memos, chats, or whatever, as long as the table has a natural array (Comma) ending, we should be fine with append.

But the Json parser can just as easily work with a memory based Qson, so the engine can parrse out to memory, then run and append against the data. It comes down to load three words, adn the lenth of the key value into sqlite3, it needs only find the next available row space,grab the key value, and return. Optimized B Trees look nice here. But is should be extremely fast, as fast as any Javascript could do, I would think. This is all very interesting to me.

No comments: