Slinq - Streaming LINQ

Source code checked in, #69709

Project Collection Service Accounts — Mon, 01 Oct 2012 22:10:56 GMT

Upgrade: New Version of LabDefaultTemplate.xaml. To upgrade your build definitions, please visit the following link: http://go.microsoft.com/fwlink/?LinkId=254563

Source code checked in, #69708

Project Collection Service Accounts — Mon, 01 Oct 2012 22:06:45 GMT

Checked in by server upgrade

Created Issue: Integrating with WCF [21976]

anup1252000 — Tue, 27 Dec 2011 04:36:52 GMT

I am finding it very difficult to integrate with WCF. Can you please help me on this.

Source code checked in, #52373

_TFSSERVICE — Wed, 28 Jul 2010 17:08:20 GMT

Checked in by server upgrade

New Post: Where do you get your streaming stock data?

bstabile — Sun, 27 Sep 2009 06:41:09 GMT

Hello,

This looks like something I would like to do also! Are you still actively working on the technique?

New Post: Where do you get your streaming stock data?

ashlar64 — Sat, 19 Sep 2009 21:06:14 GMT

Hello,

This looks like something I would like to do. I definitely will have to check this out. But I am curious as to where do you get your streaming stock quotes from?

Thanks!

Updated Wiki: Home

onovotny — Sun, 22 Jun 2008 14:13:30 GMT

Project Description
An implementation of LINQ focused on streaming data.

Slinq is a set of extension methods that implement the LINQ pattern geared for constantly changing data. Examples include sensor networks, network monitors, financial services or any other push-based systems where the client needs to display or take action on "live" data.

I've uploaded an initial prototype that demonstrates the idea; a lot more work is required to implement the various LINQ methods. Please contact me with patches if you're interested in contributing. If you have any suggestions/comments/feedback, please post in the discussion forum.

To build/run the source, you'll need Visual Studio 2008.

Design Overview

The test harness contains code that would normally be split among a data layer and client layer. The MarketDataService simulates (poorly) a stream of constantly changing market data. It runs on its own thread (a background timer) and updates the data values. In a real application, you'd have some other listener doing this based on "real" events. The Grid is a DataGridView in VirtualMode; there's currently a throttle (a winforms Timer) that takes a current snapshot of the query every n-milliseconds for a display cache. This can, and should, be further optimized in a real application.

One important thing to note is that despite the polling for snapshots, the entire query itself is not being re-run each time. The grid timer simply takes a read-lock on the stream, copies the current data into a display cache that the UI can use w/o locks and releases the lock on the query. The only time the query is currently being re-run is when the "toggle" button is pressed. That could be further enhanced to change the where clause w/o re-assembling the query. Another important note is that while references to the data elements are being copied in a few places, the underlying data itself is in memory only once. That is, if you have two queries against any given source, the data content is in the source once. Only the references are moving through the query stream.

When looking at the code, the real/interesting code is in the Slinq library in the StreamAdapter and SynchronizedBindingCollection classes. The collection classes generate events that the StreamAdapters listen for and propagate as needed. So unlike the built-in IEnumerable-based implementation where the iterators are forward-only, this implementation will respond to changing items in the source collection.

The other area of potential interest is the data model. In order to be able to join multiple pieces of data and not lose any of the data, a containment system is used. Data is stored and accessed via Schema's that define fields on the data. Multiple pieces of content can be put into a ContentSet and still be accessed directly by the schema's accessors. In this way, we can put both order data along with its joined market data into a single ContentSet. Market data is extracted using the market data schema field accessors and order data using an order schema field accessor. A future grid implementation could determine the set of schemas and available fields at the result of the query and generate the appropriate columns. Calculated columns could also be added by an adapter -- it could use a calculated schema to store a piece of calculated content within the ContentSet. The grid would read and display it seamlessly.

Note that the data model is not required in order to use Slinq, but it's an option. The only real requirement is that your type implement IEquatable in terms of some key value per record. This is important since records should be immutable. Nothing enforces this, but an immutable data type eliminates threading issues around partial reads and inconstant data. A future release might allow a key function to be specified instead of using IEquatable, but that's TBD.

Demo

Below is a screen cast of the test harness. Pressing the "Populate 50" button adds 50 random data elements. Clicking "Toggle Ask/Bid" changes the sort order to either the Bid or Ask column. In the case of Bid, it does a descending and for Ask it's ascending. While the example is a bit contrived, as you generally wouldn't query in this way, it proves that it works.

I apologize for the quality of the video; CodePlex doesn't allow you to embed a movie/flash document with the correct size, so it's an animated GIF instead. The real application is far more responsive and smooth than the video.

If you have any questions, please post them!
Thanks,
Oren Novotny

UPDATED WIKI: Home

onovotny — Fri, 22 Feb 2008 03:00:13 GMT

Design Overview

Demo

UPDATED RELEASE: 0.3.5 - .NET 3.5 RTM (Feb 22, 2008)

Fri, 22 Feb 2008 02:35:50 GMT

This build supports .NET 3.5 RTM.

* Bug fix in sorting
* From .3.3, fix race condition

RELEASED: 0.3.5 - .NET 3.5 RTM (Feb 22, 2008)

Fri, 22 Feb 2008 02:35:50 GMT

This build supports .NET 3.5 RTM.

* Bug fix in sorting
* From .3.3, fix race condition

UPDATED RELEASE: 0.3.5 - .NET 3.5 RTM (Feb 22, 2008)

Fri, 22 Feb 2008 02:30:18 GMT

This build supports .NET 3.5 RTM. Bug fix in sorting.

UPDATED RELEASE: 0.3.5 - .NET 3.5 RTM (Feb 21, 2008)

Fri, 22 Feb 2008 02:29:19 GMT

This build supports .NET 3.5 RTM. Bug fix in sorting.

UPDATED RELEASE: 0.3.4 - .NET 3.5 RTM (Feb 21, 2008)

Fri, 22 Feb 2008 01:05:39 GMT

This build supports .NET 3.5 RTM. It also contains a fix for a race condition.

UPDATED WIKI: Home

onovotny — Fri, 22 Feb 2008 00:57:15 GMT

Design Overview

Demo

UPDATED WIKI: Home

onovotny — Fri, 22 Feb 2008 00:53:40 GMT

Design Overview

The test harness contains code that would normally be split among a data layer and client layer. The MarketDataService simulates (poorly) a stream of constantly changing market data. It runs on its own thread (a background timer) and updates the data values. In a real application, you'd have some other listener doing this based on "real" events. The Grid is a DataGridView in VirtualMode; there's currently a throttle (a winforms Timer) that takes a current snapshot of the query every n-milliseconds. This can, and should, be further optimized in a real application.

One important thing to note is that despite the polling for snapshots, the entire query itself is not being re-run each time. The grid timer simply takes a read-lock on the stream, copies the current data into a buffer and releases the lock. The only time the query is currently being re-run is when the "toggle" button is pressed. That could be further enhanced to change the where clause w/o re-assembling the query. Another important note is that while references to the data elements are being copied in a few places, the underlying data itself is in memory only once. That is, if you have two queries against any given source, the data content is in the source once. Only the references are moving through the query stream.

When looking at the code, the real/interesting code is in the Slinq library in the StreamAdapter and SynchronizedBindingCollection classes. The collection classes generate events that the StreamAdapters listen for and propagate as needed. So unlike the built-in IEnumerable-based implementation where the iterators are forward-only, this implementation will respond to changing values in the source data.

The other area of potential interest is the data model. In order to be able to join multiple pieces of data and not lose any of the data, a containment system is used. Data is stored and accessed via Schema's that define fields on the data. Multiple pieces of content can be put into a ContentSet and still be accessed directly by the schema's accessors. In this way, we can put both order data along with its joined market data into a single ContentSet. Market data is extracted using the market data schema field accessors and order data using an order schema field accessor. A future grid implementation could determine the set of schemas and available fields at the result of the query and generate the appropriate columns. Calculated columns could also be added by an adapter -- it could use a calculated schema to store a piece of calculated content within the ContentSet. The grid would read and display it seamlessly.

Demo

UPDATED RELEASE: 0.3.4 - .NET 3.5 RTM (Feb 21, 2008)

Fri, 22 Feb 2008 00:52:51 GMT

This build supports .NET 3.5 RTM. It also contains a fix for a race condition.

UPDATED WIKI: Home

onovotny — Fri, 27 Jul 2007 03:00:44 GMT

Design Overview

The test harness contains code that would normally be split among a data layer and client layer. The MarketDataService simulates (poorly) a stream of constantly changing market data. It runs on its own thread (a background timer) and updates the data values. In a real application, you'd have some other listener doing this based on "real" events. The Grid is a DataGridView in VirtualMode; there's currently a throttle (a winforms Timer) that takes a current snapshot of the query every n-milliseconds. This can, and should, be further optimized in a real application.

One important thing to note is that despite the polling for snapshots, the entire query itself is not being re-run each time. The grid timer simply takes a read-lock on the stream, copies the current data into a buffer and releases the lock. The only time the query is currently being re-run is when the "toggle" button is pressed. That could be further enhanced to change the where clause w/o re-assembling the query. Another important note is that while references to the data elements are being copied in a few places, the underlying data itself is in memory only once. That is, if you have two queries against any given source, the data content is in the source once. Only the references are moving through the query stream.

When looking at the code, the real/interesting code is in the Slinq library in the StreamAdapter and SynchronizedBindingCollection classes. The collection classes generate events that the StreamAdapters listen for and propagate as needed. So unlike the built-in IEnumerable-based implementation where the iterators are forward-only, this implementation will respond to changing values in the source data.

The other area of potential interest is the data model. In order to be able to join multiple pieces of data and not lose any of the data, a containment system is used. Data is stored and accessed via Schema's that define fields on the data. Multiple pieces of content can be put into a ContentSet and still be accessed directly by the schema's accessors. In this way, we can put both order data along with its joined market data into a single ContentSet. Market data is extracted using the market data schema field accessors and order data using an order schema field accessor. A future grid implementation could determine the set of schemas and available fields at the result of the query and generate the appropriate columns. Calculated columns could also be added by an adapter -- it could use a calculated schema to store a piece of calculated content within the ContentSet. The grid would read and display it seamlessly.

Demo

UPDATED RELEASE: 0.3.2 - Fx 3.5 b2 compatibility (Jul 26, 2007)

Fri, 27 Jul 2007 02:56:23 GMT

This build supports .NET 3.5 beta 2. It also contains a few refactorings.

UPDATED RELEASE: Initial Release 0.3.1 (Jun 02, 2007)

Fri, 27 Jul 2007 02:55:39 GMT

Proof of concept that demonstrates LINQ against a streaming data source.

UPDATED RELEASE: Beta 2 compatibility 0.3.2 (Jul 26, 2007)

Fri, 27 Jul 2007 02:55:17 GMT

This build supports .NET 3.5 beta 2. It also contains a few refactorings.