Project Description
An implementation of LINQ focused on streaming data.

Slinq is a set of extension methods that implement the LINQ pattern geared for constantly changing data. Examples include sensor networks, network monitors, financial services or any other push-based systems where the client needs to display or take action on "live" data.

I've uploaded an initial prototype that demonstrates the idea; a lot more work is required to implement the various LINQ methods. Please contact me with patches if you're interested in contributing. If you have any suggestions/comments/feedback, please post in the discussion forum.

To build/run the source, you'll need Visual Studio 2008.

Design Overview

The test harness contains code that would normally be split among a data layer and client layer. The MarketDataService simulates (poorly) a stream of constantly changing market data. It runs on its own thread (a background timer) and updates the data values. In a real application, you'd have some other listener doing this based on "real" events. The Grid is a DataGridView in VirtualMode; there's currently a throttle (a winforms Timer) that takes a current snapshot of the query every n-milliseconds for a display cache. This can, and should, be further optimized in a real application.

One important thing to note is that despite the polling for snapshots, the entire query itself is not being re-run each time. The grid timer simply takes a read-lock on the stream, copies the current data into a display cache that the UI can use w/o locks and releases the lock on the query. The only time the query is currently being re-run is when the "toggle" button is pressed. That could be further enhanced to change the where clause w/o re-assembling the query. Another important note is that while references to the data elements are being copied in a few places, the underlying data itself is in memory only once. That is, if you have two queries against any given source, the data content is in the source once. Only the references are moving through the query stream.

When looking at the code, the real/interesting code is in the Slinq library in the StreamAdapter and SynchronizedBindingCollection classes. The collection classes generate events that the StreamAdapters listen for and propagate as needed. So unlike the built-in IEnumerable-based implementation where the iterators are forward-only, this implementation will respond to changing items in the source collection.

The other area of potential interest is the data model. In order to be able to join multiple pieces of data and not lose any of the data, a containment system is used. Data is stored and accessed via Schema's that define fields on the data. Multiple pieces of content can be put into a ContentSet and still be accessed directly by the schema's accessors. In this way, we can put both order data along with its joined market data into a single ContentSet. Market data is extracted using the market data schema field accessors and order data using an order schema field accessor. A future grid implementation could determine the set of schemas and available fields at the result of the query and generate the appropriate columns. Calculated columns could also be added by an adapter -- it could use a calculated schema to store a piece of calculated content within the ContentSet. The grid would read and display it seamlessly.

Note that the data model is not required in order to use Slinq, but it's an option. The only real requirement is that your type implement IEquatable in terms of some key value per record. This is important since records should be immutable. Nothing enforces this, but an immutable data type eliminates threading issues around partial reads and inconstant data. A future release might allow a key function to be specified instead of using IEquatable, but that's TBD.

Demo

Below is a screen cast of the test harness. Pressing the "Populate 50" button adds 50 random data elements. Clicking "Toggle Ask/Bid" changes the sort order to either the Bid or Ask column. In the case of Bid, it does a descending and for Ask it's ascending. While the example is a bit contrived, as you generally wouldn't query in this way, it proves that it works.

I apologize for the quality of the video; CodePlex doesn't allow you to embed a movie/flash document with the correct size, so it's an animated GIF instead. The real application is far more responsive and smooth than the video.

slinq.gif

If you have any questions, please post them!
Thanks,
Oren Novotny

Last edited Jun 22, 2008 at 3:13 PM by onovotny, version 19