Microsoft announces open source data processing engine Trill
Microsoft recently opened up the data processing engine Trill, which analyzes trillions of events a day. The current processing of large amounts of data per millisecond is becoming a common business requirement. The Microsoft open source Trill, which is said to handle up to billions of events per second, combines multimodal analysis support with a host of other features. It has the following characteristics:
- As a single-node engine library, any .NET application, service, or platform can easily use Trill and start processing queries.
- A temporal query language allows users to express complex queries over real-time and/or offline data sets.
- Trill’s high performance across its intended usage scenarios means users get results with incredible speed and low latency. For example, filters operate at memory bandwidth speeds up to several billions of events per second, while grouped aggregates operate at 10 to 100 million events per second.
The engine is versatile enough to handle real-time and historical data, and only a few open source tools have the same capabilities.
Trill began as a research project at Microsoft Research in 2012 and was extensively described in research papers such as VLDB and IEEE Data Engineering Bulletin. Trill was originally derived from Microsoft’s previous service, StreamInsight, a powerful platform that allows developers to develop and deploy complex event-processing applications. Both systems are based on an extended query and data model that uses a time component to extend the relational model.
For users, using Trill is the same as using any .NET library, so there is no need to leave the .NET environment. Users can embed Trill into a variety of distributed processing infrastructures, such as streaming versions of Orleans and Microsoft’s SCOPE data processing infrastructure.
Trill supports internal applications and external services, covering thousands of developers, and has provided many powerful streaming services, including:
- Financial Fabric
- Bing Ads
- Azure Stream Analytics
- Halo
Project address is available here.
