Jay Taylor's notes
back to listing indexEsper Reference
[web search]- Preface
- 1. Technology Overview
- 2. Event Representations
- 2.1. Event Underlying Java Objects
- 2.2. Event Properties
- 2.3. Dynamic Event Properties
- 2.4. Fragment and Fragment Type
- 2.5. Plain-Old Java Object Events
- 2.6. java.util.Map Events
- 2.7. Object-array (Object[]) Events
- 2.8. org.w3c.dom.Node XML Events
- 2.9. Additional Event Representations
- 2.10. Updating, Merging and Versioning Events
- 2.11. Coarse-Grained Events
- 2.12. Event Objects Instantiated and Populated by Insert Into
- 2.13. Comparing Event Representations
- 3. Processing Model
- 4. Context and Context Partitions
- 5. EPL Reference: Clauses
- 5.1. EPL Introduction
- 5.2. EPL Syntax
- 5.3. Choosing Event Properties And Events: the Select Clause
- 5.3.1. Choosing all event properties: select *
- 5.3.2. Choosing specific event properties
- 5.3.3. Expressions
- 5.3.4. Renaming event properties
- 5.3.5. Choosing event properties and events in a join
- 5.3.6. Choosing event properties and events from a pattern
- 5.3.7. Selecting insert and remove stream events
- 5.3.8. Qualifying property names and stream names
- 5.3.9. Select Distinct
- 5.3.10. Transposing an Expression Result to a Stream
- 5.3.11. Selecting EventBean instead of Underlying Event
- 5.4. Specifying Event Streams: the From Clause
- 5.5. Specifying Search Conditions: the Where Clause
- 5.6. Aggregates and grouping: the Group-by Clause and the Having Clause
- 5.6.1. Using aggregate functions
- 5.6.2. Organizing statement results into groups: the Group-by clause
- 5.6.3. Using Group-By with Rollup, Cube and Grouping Sets
- 5.6.4. Selecting groups of events: the Having clause
- 5.6.5. How the stream filter, Where, Group By and Having clauses interact
- 5.6.6. Comparing Keyed Segmented Context, the Group By clause and the std:groupwin view
- 5.7. Stabilizing and Controlling Output: the Output Clause
- 5.8. Sorting Output: the Order By Clause
- 5.9. Limiting Row Count: the Limit Clause
- 5.10. Merging Streams and Continuous Insertion: the Insert Into Clause
- 5.10.1. Transposing a Property To a Stream
- 5.10.2. Merging Streams By Event Type
- 5.10.3. Merging Disparate Types of Events: Variant Streams
- 5.10.4. Decorated Events
- 5.10.5. Event as a Property
- 5.10.6. Instantiating and Populating an Underlying Event Object
- 5.10.7. Transposing an Expression Result
- 5.10.8. Select-Clause Expression And Inserted-Into Column Event Type
- 5.11. Subqueries
- 5.12. Joining Event Streams
- 5.13. Accessing Relational Data via SQL
- 5.13.1. Joining SQL Query Results
- 5.13.2. SQL Query and the EPL Where Clause
- 5.13.3. Outer Joins With SQL Queries
- 5.13.4. Using Patterns to Request (Poll) Data
- 5.13.5. Polling SQL Queries via Iterator
- 5.13.6. JDBC Implementation Overview
- 5.13.7. Oracle Drivers and No-Metadata Workaround
- 5.13.8. SQL Input Parameter and Column Output Conversion
- 5.13.9. SQL Row POJO Conversion
- 5.14. Accessing Non-Relational Data via Method Invocation
- 5.15. Declaring an Event Type: Create Schema
- 5.16. Splitting and Duplicating Streams
- 5.17. Variables and Constants
- 5.18. Declaring Global Expressions, Aliases And Scripts: Create Expression
- 5.19. Contained-Event Selection
- 5.20. Updating an Insert Stream: the Update IStream Clause
- 5.21. Controlling Event Delivery : The For Clause
- 6. EPL Reference: Named Windows And Tables
- 6.1. Overview
- 6.2. Named Window Usage
- 6.3. Table Usage
- 6.4. Triggered Select: the On Select clause
- 6.5. Triggered Select+Delete: the On Select Delete clause
- 6.6. Updating Data: the On Update clause
- 6.7. Deleting Data: the On Delete clause
- 6.8. Triggered Upsert using the On-Merge Clause
- 6.9. Explicitly Indexing Named Windows and Tables
- 6.10. Using Fire-And-Forget Queries with Named Windows and Tables
- 6.11. Versioning and Revision Event Type Use with Named Windows
- 6.12. Events As Property
- 7. EPL Reference: Patterns
- 8. EPL Reference: Match Recognize
- 8.1. Overview
- 8.2. Comparison of Match Recognize and EPL Patterns
- 8.3. Syntax
- 8.4. Pattern and Pattern Operators
- 8.4.1. Operator Precedence
- 8.4.2. Concatenation
- 8.4.3. Alternation
- 8.4.4. Quantifiers Overview
- 8.4.5. Variables Can be Singleton or Group
- 8.4.6. Eliminating Duplicate Matches
- 8.4.7. Greedy Or Reluctant
- 8.4.8. Quantifier - One Or More (+ and +?)
- 8.4.9. Quantifier - Zero Or More (* and *?)
- 8.4.10. Quantifier - Zero Or One (? and ??)
- 8.5. Define Clause
- 8.6. Measure Clause
- 8.7. Datawindow-Bound
- 8.8. Interval
- 8.9. Interval-Or-Terminated
- 8.10. Use with Different Event Types
- 8.11. Limitations
- 9. EPL Reference: Operators
- 9.1. Arithmetic Operators
- 9.2. Logical And Comparison Operators
- 9.3. Concatenation Operators
- 9.4. Binary Operators
- 9.5. Array Definition Operator
- 9.6. Dot Operator
- 9.7. The 'in' Keyword
- 9.8. The 'between' Keyword
- 9.9. The 'like' Keyword
- 9.10. The 'regexp' Keyword
- 9.11. The 'any' and 'some' Keywords
- 9.12. The 'all' Keyword
- 9.13. The 'new' Keyword
- 10. EPL Reference: Functions
- 10.1. Single-row Function Reference
- 10.1.1. The Case Control Flow Function
- 10.1.2. The Cast Function
- 10.1.3. The Coalesce Function
- 10.1.4. The Current_Timestamp Function
- 10.1.5. The Exists Function
- 10.1.6. The Grouping Function
- 10.1.7. The Grouping_Id Function
- 10.1.8. The Instance-Of Function
- 10.1.9. The Istream Function
- 10.1.10. The Min and Max Functions
- 10.1.11. The Previous Function
- 10.1.12. The Previous-Tail Function
- 10.1.13. The Previous-Window Function
- 10.1.14. The Previous-Count Function
- 10.1.15. The Prior Function
- 10.1.16. The Type-Of Function
- 10.2. Aggregation Functions
- 10.3. User-Defined Functions
- 10.4. Select-Clause transpose Function
- 11. EPL Reference: Enumeration Methods
- 11.1. Overview
- 11.2. Example Events
- 11.3. How to Use
- 11.4. Inputs
- 11.4.1. Subquery Results
- 11.4.2. Named Window
- 11.4.3. Table
- 11.4.4. Event Property
- 11.4.5. Event Aggregation Function
- 11.4.6. prev, prevwindow and prevtail Single-Row Functions as Input
- 11.4.7. Single-Row Function, User-Defined Function and Enum Types
- 11.4.8. Declared Expression
- 11.4.9. Variables
- 11.4.10. Substitution Parameters
- 11.4.11. Match-Recognize Group Variable
- 11.4.12. Pattern Repeat and Repeat-Until Operators
- 11.5. Example
- 11.6. Reference
- 11.6.1. Aggregate
- 11.6.2. AllOf
- 11.6.3. AnyOf
- 11.6.4. Average
- 11.6.5. CountOf
- 11.6.6. DistinctOf
- 11.6.7. Except
- 11.6.8. FirstOf
- 11.6.9. GroupBy
- 11.6.10. Intersect
- 11.6.11. LastOf
- 11.6.12. LeastFrequent
- 11.6.13. Max
- 11.6.14. MaxBy
- 11.6.15. Min
- 11.6.16. MinBy
- 11.6.17. MostFrequent
- 11.6.18. OrderBy and OrderByDesc
- 11.6.19. Reverse
- 11.6.20. SelectFrom
- 11.6.21. SequenceEqual
- 11.6.22. SumOf
- 11.6.23. Take
- 11.6.24. TakeLast
- 11.6.25. TakeWhile
- 11.6.26. TakeWhileLast
- 11.6.27. ToMap
- 11.6.28. Union
- 11.6.29. Where
- 12. EPL Reference: Date-Time Methods
- 12.1. Overview
- 12.2. How to Use
- 12.3. Calendar and Formatting Reference
- 12.4. Interval Algebra Reference
- 12.4.1. Examples
- 12.4.2. Interval Algebra Parameters
- 12.4.3. Performance
- 12.4.4. Limitations
- 12.4.5. After
- 12.4.6. Before
- 12.4.7. Coincides
- 12.4.8. During
- 12.4.9. Finishes
- 12.4.10. Finished By
- 12.4.11. Includes
- 12.4.12. Meets
- 12.4.13. Met By
- 12.4.14. Overlaps
- 12.4.15. Overlapped By
- 12.4.16. Starts
- 12.4.17. Started By
- 13. EPL Reference: Views
- 13.1. A Note on View Parameters
- 13.2. Data Window Views
- 13.2.1. Length window (win:length)
- 13.2.2. Length batch window (win:length_batch)
- 13.2.3. Time window (win:time)
- 13.2.4. Externally-timed window (win:ext_timed)
- 13.2.5. Time batch window (win:time_batch)
- 13.2.6. Externally-timed batch window (win:ext_timed_batch)
- 13.2.7. Time-Length combination batch window (win:time_length_batch)
- 13.2.8. Time-Accumulating window (win:time_accum)
- 13.2.9. Keep-All window (win:keepall)
- 13.2.10. First Length (win:firstlength)
- 13.2.11. First Time (win:firsttime)
- 13.2.12. Expiry Expression (win:expr)
- 13.2.13. Expiry Expression Batch (win:expr_batch)
- 13.3. Standard view set
- 13.4. Statistics views
- 13.5. Extension View Set
- 14. EPL Reference: Data Flow
- 15. API Reference
- 15.1. API Overview
- 15.2. The Service Provider Interface
- 15.3. The Administrative Interface
- 15.4. The Runtime Interface
- 15.5. On-Demand Fire-And-Forget Query Execution
- 15.6. Event and Event Type
- 15.7. Engine Threading and Concurrency
- 15.8. Controlling Time-Keeping
- 15.9. Time Resolution
- 15.10. Service Isolation
- 15.11. Exception Handling
- 15.12. Condition Handling
- 15.13. Statement Object Model
- 15.14. Prepared Statement and Substitution Parameters
- 15.15. Engine and Statement Metrics Reporting
- 15.16. Event Rendering to XML and JSON
- 15.17. Plug-in Loader
- 15.18. Interrogating EPL Annotations
- 15.19. Context Partition Selection
- 15.20. Context Partition Administration
- 15.21. Test and Assertion Support
- 16. Configuration
- 16.1. Programmatic Configuration
- 16.2. Configuration via XML File
- 16.3. XML Configuration File
- 16.4. Configuration Items
- 16.4.1. Events represented by Java Classes
- 16.4.2. Events represented by java.util.Map
- 16.4.3. Events represented by Object[] (Object-array)
- 16.4.4. Events represented by org.w3c.dom.Node
- 16.4.5. Events represented by Plug-in Event Representations
- 16.4.6. Class and package imports
- 16.4.7. Cache Settings for From-Clause Method Invocations
- 16.4.8. Variables
- 16.4.9. Relational Database Access
- 16.4.10. Engine Settings related to Concurrency and Threading
- 16.4.11. Engine Settings related to Event Metadata
- 16.4.12. Engine Settings related to View Resources
- 16.4.13. Engine Settings related to Logging
- 16.4.14. Engine Settings related to Variables
- 16.4.15. Engine Settings related to Patterns
- 16.4.16. Engine Settings related to Scripts
- 16.4.17. Engine Settings related to Stream Selection
- 16.4.18. Engine Settings related to Time Source
- 16.4.19. Engine Settings related to JMX Metrics
- 16.4.20. Engine Settings related to Metrics Reporting
- 16.4.21. Engine Settings related to Language and Locale
- 16.4.22. Engine Settings related to Expression Evaluation
- 16.4.23. Engine Settings related to Execution of Statements
- 16.4.24. Engine Settings related to Exception Handling
- 16.4.25. Engine Settings related to Condition Handling
- 16.4.26. Revision Event Type
- 16.4.27. Variant Stream
- 16.5. Type Names
- 16.6. Runtime Configuration
- 16.7. Logging Configuration
- 17. Development Lifecycle
- 18. Integration and Extension
- 19. Script Support
- 20. Examples, Tutorials, Case Studies
- 20.1. Examples Overview
- 20.2. Running the Examples
- 20.3. AutoID RFID Reader
- 20.4. Runtime Configuration
- 20.5. JMS Server Shell and Client
- 20.6. Market Data Feed Monitor
- 20.7. OHLC Plug-in View
- 20.8. Transaction 3-Event Challenge
- 20.9. Self-Service Terminal
- 20.10. Assets Moving Across Zones - An RFID Example
- 20.11. StockTicker
- 20.12. MatchMaker
- 20.13. Named Window Query
- 20.14. Sample Virtual Data Window
- 20.15. Sample Cycle Detection
- 20.16. Quality of Service
- 20.17. Trivia Geeks Club
- 21. Performance
- 21.1. Performance Results
- 21.2. Performance Tips
- 21.2.1. Understand how to tune your Java virtual machine
- 21.2.2. Input and Output Bottlenecks
- 21.2.3. Theading
- 21.2.4. Select the underlying event rather than individual fields
- 21.2.5. Prefer stream-level filtering over where-clause filtering
- 21.2.6. Reduce the use of arithmetic in expressions
- 21.2.7. Remove Unneccessary Constructs
- 21.2.8. End Pattern Sub-Expressions
- 21.2.9. Consider using EventPropertyGetter for fast access to event properties
- 21.2.10. Consider casting the underlying event
- 21.2.11. Turn off logging and audit
- 21.2.12. Disable view sharing
- 21.2.13. Tune or disable delivery order guarantees
- 21.2.14. Use a Subscriber Object to Receive Events
- 21.2.15. Consider Data Flows
- 21.2.16. High-Arrival-Rate Streams and Single Statements
- 21.2.17. Subqueries versus Joins And Where-clause And Data Windows
- 21.2.18. Patterns and Pattern Sub-Expression Instances
- 21.2.19. Pattern Sub-Expression Instance Versus Data Window Use
- 21.2.20. The Keep-All Data Window
- 21.2.21. Statement Design for Reduced Memory Consumption - Diagnosing OutOfMemoryError
- 21.2.22. Performance, JVM, OS and hardware
- 21.2.23. Consider using Hints
- 21.2.24. Optimizing Stream Filter Expressions
- 21.2.25. Statement and Engine Metric Reporting
- 21.2.26. Expression Evaluation Order and Early Exit
- 21.2.27. Large Number of Threads
- 21.2.28. Filter Evaluation Tuning
- 21.2.29. Context Partition Related Information
- 21.2.30. Prefer Constant Variables over Non-Constant Variables
- 21.2.31. Prefer Object-array Events
- 21.2.32. Composite or Compound Keys
- 21.2.33. Notes on Query Planning
- 21.2.34. Query Planning Expression Analysis Hints
- 21.2.35. Query Planning Index Hints
- 21.2.36. Measuring Throughput
- 21.2.37. Do not create the same EPL Statement X times
- 21.2.38. Comparing Single-Threaded and Multi-Threaded Performance
- 21.2.39. Incremental Versus Recomputed Aggregation for Named Window Events
- 21.2.40. When Does Memory Get Released
- 21.2.41. Measure throughput of non-matches as well as matches
- 21.3. Using the performance kit
- 22. References
- A. Output Reference and Samples
- A.1. Introduction and Sample Data
- A.2. Output for Un-aggregated and Un-grouped Queries
- A.3. Output for Fully-aggregated and Un-grouped Queries
- A.4. Output for Aggregated and Un-grouped Queries
- A.5. Output for Fully-aggregated and Grouped Queries
- A.6. Output for Aggregated and Grouped Queries
- A.7. Output for Fully-Aggregated, Grouped Queries With Rollup
- B. Reserved Keywords
- Index
It is assumed that the reader is familiar with the Java programming language.
If you are new to Esper, please follow these steps:
Read Section 1.1, “Introduction to CEP and event series analysis” if you are new to CEP and ESP (complex event processing, event stream processing)
Read Chapter 2, Event Representations that explains the different ways of representing events to Esper
Read Chapter 3, Processing Model to gain insight into EPL continuous query results
Read Section 5.1, “EPL Introduction” for an introduction to event stream processing via EPL
Read Section 7.1, “Event Pattern Overview” for an overview over event patterns
Read Section 8.1, “Overview” for an overview over event patterns using the match recognize syntax.
Then glance over the examples Section 20.1, “Examples Overview”
Finally to test drive Esper performance, read Chapter 21, Performance
Business process management and automation (process monitoring, BAM, reporting exceptions)
Finance (algorithmic trading, fraud detection, risk management)
Network and application monitoring (intrusion detection, SLA monitoring)
Sensor network applications (RFID reading, scheduling and control of fabrication lines, air traffic)
The Esper engine was designed to make it easier to build and extend CEP applications.
- 2.1. Event Underlying Java Objects
- 2.2. Event Properties
- 2.3. Dynamic Event Properties
- 2.4. Fragment and Fragment Type
- 2.5. Plain-Old Java Object Events
- 2.6. java.util.Map Events
- 2.7. Object-array (Object[]) Events
- 2.8. org.w3c.dom.Node XML Events
- 2.9. Additional Event Representations
- 2.10. Updating, Merging and Versioning Events
- 2.11. Coarse-Grained Events
- 2.12. Event Objects Instantiated and Populated by Insert Into
- 2.13. Comparing Event Representations
This section outlines the different means to model and represent events.
Esper uses the term event type to describe the type information available for an event representation.
Your application may configure predefined event types at startup time or dynamically add event types at runtime via API or EPL syntax. See Section 16.4, “Configuration Items” for startup-time configuration and Section 15.3.7, “Runtime Configuration” for the runtime configuration API.
The EPL create schema
syntax allows declaring an event type at runtime using EPL, see Section 5.15, “Declaring an Event Type: Create Schema”.
In Section 15.6, “Event and Event Type” we explain how an event type becomes visible in EPL statements and output events delivered by the engine.
An event is an immutable record of a past occurrence of an action or state change. Event properties capture the state information for an event.
In Esper, an event can be represented by any of the following underlying Java objects:
Table 2.1. Event Underlying Java Objects
Java Class | Description |
---|---|
java.lang.Object | Any Java POJO (plain-old java object) with getter methods following JavaBean conventions; Legacy Java classes not following JavaBean conventions can also serve as events . |
java.util.Map | Map events are implementations of the java.util.Map interface where each map entry is a propery value. |
Object[] (array of object) | Object-array events are arrays of objects (type Object[] ) where each array element is a property value. |
org.w3c.dom.Node | XML document object model (DOM). |
org.apache.axiom.om.OMDocument or OMElement | XML - Streaming API for XML (StAX) - Apache Axiom (provided by EsperIO package). |
Application classes | Plug-in event representation via the extension API. |
Esper provides multiple choices for representing an event. There is no absolute need for you to create new Java classes to represent an event.
Event representations have the following in common:
All event representations support nested, indexed and mapped properties (aka. property expression), as explained in more detail below. There is no limitation to the nesting level.
All event representations provide event type metadata. This includes type metadata for nested properties.
All event representations allow transposing the event itself and parts of all of its property graph into new events. The term transposing refers to selecting the event itself or event properties that are themselves nestable property graphs, and then querying the event's properties or nested property graphs in further statements. The Apache Axiom event representation is an exception and does not currently allow transposing event properties but does allow transposing the event itself.
The Java object, Map and Object-array representations allow supertypes.
The API behavior for all event representations is the same, with minor exceptions noted in this chapter.
The benefits of multiple event representations are:
For applications that already have events in one of the supported representations, there is no need to transform events into a Java object before processing.
Event representations are exchangeable, reducing or eliminating the need to change statements when the event representation changes.
Event representations are interoperable, allowing all event representations to interoperate in same or different statements.
The choice makes its possible to consciously trade-off performance, ease-of-use, the ability to evolve and effort needed to import or externalize events and use existing event type metadata.
Event properties capture the state information for an event. Event properties be simple as well as indexed, mapped and nested event properties. The table below outlines the different types of properties and their syntax in an event expression. This syntax allows statements to query deep JavaBean objects graphs, XML structures and Map events.
Table 2.2. Types of Event Properties
Type | Description | Syntax | Example |
---|---|---|---|
Simple | A property that has a single value that may be retrieved. | name | sensorId |
Indexed | An indexed property stores an ordered collection of objects (all of the same type) that can be individually accessed by an integer-valued, non-negative index (or subscript). | name[index] | sensor[0] |
Mapped | A mapped property stores a keyed collection of objects (all of the same type). | name('key') | sensor('light') |
Nested | A nested property is a property that lives within another property of an event. | name.nestedname | sensor.value |
Combinations are also possible. For example, a valid combination could be person.address('home').street[0]
.
You may use any expression as a mapped property key or indexed property index by putting the expression within parenthesis after the mapped or index property name. Please find examples below.
select part1\.part2 from MyEvent
select `order`, price as `price.for.goods` from Quote
select `candidate book` , `children's books`[0], `book select`('isbn') from MyEventType
Assume a class declares these properties (getters not shown for brevity):
public class MyEventType { String myMapKey; int myIndexValue; int myInnerIndexValue; Map<String, InnerType> innerTypesMap; // mapped property InnerType[] innerTypesArray; // indexed property } public class InnerType { String name; int[] ids; }
A sample EPL statement demonstrating expressions as map keys or indexes is:
select innerTypesMap('somekey'), // returns map value for 'somekey' innerTypesMap(myMapKey), // returns map value for myMapKey value (an expression) innerTypesArray[1], // returns array value at index 1 innerTypesArray(myIndexValue) // returns array value at index myIndexValue (an expression) from MyEventType
The dot-operator can be used to access methods on the value objects returned by the mapped or indexed properties. By using the dot-operator the syntax follows the chained method invocation described at Section 9.6, “Dot Operator”.
A sample EPL statement demonstrating the dot-operator as well as expressions as map keys or indexes is:
select innerTypesMap('somekey').ids[1], innerTypesMap(myMapKey).getIds(myIndexValue), innerTypesArray[1].ids[2], innerTypesArray(myIndexValue).getIds(myInnerIndexValue) from MyEventType
Please note the following limitations:
The square brackets-syntax for indexed properties does now allow expressions and requires a constant index value.
When using the dot-operator with mapped or indexed properties that have expressions as map keys or indexes you must follow the chained method invocation syntax.
Dynamic (unchecked) properties are event properties that need not be known at statement compilation time. Such properties are resolved during runtime: they provide duck typing functionality.
The idea behind dynamic properties is that for a given underlying event representation we don't always know all properties in advance. An underlying event may have additional properties that are not known at statement compilation time, that we want to query on. The concept is especially useful for events that represent rich, object-oriented domain models.
The syntax of dynamic properties consists of the property name and a question mark. Indexed, mapped and nested properties can also be dynamic properties:
Table 2.3. Types of Event Properties
Type | Syntax |
---|---|
Dynamic Simple | name? |
Dynamic Indexed | name[index]? |
Dynamic Mapped | name('key')? |
Dynamic Nested | name?.nestedPropertyName |
Dynamic properties always return the java.lang.Object
type. Also, dynamic properties return a null
value if the dynamic property does not exist on events processed at runtime.
As an example, consider an OrderEvent event that provides an "item" property. The "item" property is of type Object
and holds a reference to an instance of either a Service or Product.
Assume that both Service and Product classes provide a property named "price". Via a dynamic property we can specify a query that obtains the price property from either object (Service or Product):
select item.price? from OrderEvent
As a second example, assume that the Service class contains a "serviceName" property that the Product class does not possess. The following query returns the value of the "serviceName" property for Service objects. It returns a null
-value for Product objects that do not have the "serviceName" property:
select item.serviceName? from OrderEvent
Consider the case where OrderEvent has multiple implementation classes, some of which have a "timestamp" property. The next query returns the timestamp property of those implementations of the OrderEvent interface that feature the property:
select timestamp? from OrderEvent
The query as above returns a single column named "timestamp?" of type Object
.
When dynamic properties are nested, then all properties under the dynamic property are also considered dynamic properties. In the below example the query asks for the "direction" property of the object returned by the "detail" dynamic property:
select detail?.direction from OrderEvent
Above is equivalent to:
select detail?.direction? from OrderEvent
The functions that are often useful in conjunction with dynamic properties are:
The
cast
function casts the value of a dynamic property (or the value of an expression) to a given type.The
exists
function checks whether a dynamic property exists. It returnstrue
if the event has a property of that name, or false if the property does not exist on that event.The
instanceof
function checks whether the value of a dynamic property (or the value of an expression) is of any of the given types.The
typeof
function returns the string type name of a dynamic property.
Dynamic event properties work with all event representations outlined next: Java objects, Map-based, Object-array-based and XML DOM-based events.
Fragments and type metadata can allow your application to navigate composite events without the need for using the Java reflection API and reducing the coupling to the underlying event representation. The API is further described in Section 15.6, “Event and Event Type”.
Plain-old Java object events are object instances that expose event properties through JavaBeans-style getter methods. Events classes or interfaces do not have to be fully compliant to the JavaBean specification; however for the Esper engine to obtain event properties, the required JavaBean getter methods must be present or an accessor-style and accessor-methods may be defined via configuration.
Esper supports JavaBeans-style event classes that extend a superclass or implement one or more interfaces. Also, Esper event pattern and EPL statements can refer to Java interface classes and abstract classes.
Classes that represent events should be made immutable. As events are recordings of a state change or action that occurred in the past, the relevant event properties should not be changeable. However this is not a hard requirement and the Esper engine accepts events that are mutable as well.
The hashCode
and equals
methods do not need to be implemented. The implementation of these methods by a Java event class does not affect the behavior of the engine in any way.
Please see Chapter 16, Configuration on options for naming event types represented by Java object event classes. Java classes that do not follow JavaBean conventions, such as legacy Java classes that expose public fields, or methods not following naming conventions, require additional configuration. Via configuration it is also possible to control case sensitivity in property name resolution. The relevant section in the chapter on configuration is Section 16.4.1.3, “Non-JavaBean and Legacy Java Event Classes”.
public class NewEmployeeEvent { public String getFirstName(); public Address getAddress(String type); public Employee getSubordinate(int index); public Employee[] getAllSubordinates(); }
every NewEmployeeEvent(firstName='myName') every NewEmployeeEvent(address('home').streetName='Park Avenue') every NewEmployeeEvent(subordinate[0].name='anotherName') every NewEmployeeEvent(allSubordinates[1].name='thatName') every NewEmployeeEvent(subordinate[0].address('home').streetName='Water Street')
select firstName, address('work'), subordinate[0].name, subordinate[1].name from NewEmployeeEvent where address('work').streetName = 'Park Ave'
An example event that has properties that are parameterized types is:
public class NewEmployeeEvent { public String getName(); public Iterable<EducationHistory> getEducation(); public Map<String, Address> getAddresses(); }
A sample of valid property expressions for this event is shown next:
select name, education, education[0].date, addresses('home').street from NewEmployeeEvent
public class MyEvent { private Map props = new HashMap(); private Object[] array = new Object[10]; public void setProps(String name, Object value) { props.put(name, value); } public void setArray(int index, Object value) { array[index] = value; } // ... also provide regular JavaBean getters and setters for all properties
This sample statement updates mapped and indexed property values:
update istream MyEventStream set props('key') = 'abc', array[2] = 100
In order to use Map
events, the event type name and property names and types must be made known to the engine via Configuration or create schema
EPL syntax. Please see examples in Section 5.15, “Declaring an Event Type: Create Schema” and Section 16.4.2, “Events represented by java.util.Map”.
The code snippet below defines a Map event type, creates a Map event and sends the event into the engine. The sample defines the CarLocUpdateEvent
event type via runtime configuration interface (create schema
or static configuration could have been used instead).
// Define CarLocUpdateEvent event type (example for runtime-configuration interface) Map<String, Object> def = new HashMap<String, Object>; def.put("carId", String.class); def.put("direction", int.class); epService.getEPAdministrator().getConfiguration(). addEventType("CarLocUpdateEvent", def);
The CarLocUpdateEvent
can now be used in a statement:
select carId from CarLocUpdateEvent.win:time(1 min) where direction = 1
// Create a CarLocUpdateEvent event and send it into the engine for processing Map<String, Object> event = new HashMap<String, Object>(); event.put("carId", carId); event.put("direction", direction); epRuntime.sendEvent(event, "CarLocUpdateEvent");
The engine can also query Java objects as values in a Map
event via the nested property syntax. Thus Map
events can be used to
aggregate multiple data structures into a single event and query the composite information in a convenient way. The example below demonstrates a Map
event with a transaction and an account object.
Map event = new HashMap(); event.put("txn", txn); event.put("account", account); epRuntime.sendEvent(event, "TxnEvent");
An example statement could look as follows.
select account.id, account.rate * txn.amount from TxnEvent.win:time(60 sec) group by account.id
epService.getEPAdministrator().getConfiguration(). addEventType("AccountUpdate", accountUpdateDef, new String[] {"BaseUpdate"});
// Receive BaseUpdate and any subtypes including subtypes of subtypes select * from BaseUpdate
See Section 16.4.2, “Events represented by java.util.Map” for more information on configuring Map event types.
Map<String, Object> updatedFieldDef = new HashMap<String, Object>(); updatedFieldDef.put("name", String.class); updatedFieldDef.put("addressLine1", String.class); updatedFieldDef.put("history", UpdateHistory.class); epService.getEPAdministrator().getConfiguration(). addEventType("UpdatedFieldType", updatedFieldDef); Map<String, Object> accountUpdateDef = new HashMap<String, Object>(); accountUpdateDef.put("accountId", long.class); accountUpdateDef.put("fields", "UpdatedFieldType"); // the latter can also be: accountUpdateDef.put("fields", updatedFieldDef); epService.getEPAdministrator().getConfiguration(). addEventType("AccountUpdate", accountUpdateDef);
The next code snippet populates a sample event and sends the event into the engine:
Map<String, Object> updatedField = new HashMap<String, Object>(); updatedField.put("name", "Joe Doe"); updatedField.put("addressLine1", "40 Popular Street"); updatedField.put("history", new UpdateHistory()); Map<String, Object> accountUpdate = new HashMap<String, Object>(); accountUpdate.put("accountId", 10009901); accountUpdate.put("fields", updatedField); epService.getEPRuntime().sendEvent(accountUpdate, "AccountUpdate");
Last, a sample query to interrogate AccountUpdate
events is as follows:
select accountId, fields.name, fields.addressLine1, fields.history.lastUpdate from AccountUpdate
Map<String, Object> sale = new HashMap<String, Object>(); sale.put("userids", int[].class); sale.put("salesPersons", SalesPerson[].class); sale.put("items", "OrderItem[]"); // The property type is the name itself appended by [] epService.getEPAdministrator().getConfiguration(). addEventType("SaleEvent", sale);
The three properties that the above example declares are:
The next EPL statement is a sample query asking for property values held by arrays:
select userids[0], salesPersons[1].name, items[1], items[1].price.amount from SaleEvent
In order to use Object[]
(object-array) events, the event type name and property names and types, in a well-defined order that must match object-array event properties, must be made known to the engine via configuration or create schema
EPL syntax. Please see examples in Section 5.15, “Declaring an Event Type: Create Schema” and Section 16.4.3, “Events represented by Object[] (Object-array)”.
The code snippet below defines an Object-array event type, creates an Object-array event and sends the event into the engine. The sample defines the CarLocUpdateEvent
event type via the runtime configuration interface (create schema
or static configuration could have been used instead).
// Define CarLocUpdateEvent event type (example for runtime-configuration interface) String[] propertyNames = {"carId", "direction"}; // order is important Object[] propertyTypes = {String.class, int.class}; // type order matches name order epService.getEPAdministrator().getConfiguration(). addEventType("CarLocUpdateEvent", propertyNames, propertyTypes);
The CarLocUpdateEvent
can now be used in a statement:
select carId from CarLocUpdateEvent.win:time(1 min) where direction = 1
// Send an event Object[] event = {carId, direction}; epRuntime.sendEvent(event, "CarLocUpdateEvent");
The engine can also query Java objects as values in an Object[]
event via the nested property syntax. Thus Object[]
events can be used to
aggregate multiple data structures into a single event and query the composite information in a convenient way. The example below demonstrates a Object[]
event with a transaction and an account object.
epRuntime.sendEvent(new Object[] {txn, account}, "TxnEvent");
An example statement could look as follows:
select account.id, account.rate * txn.amount from TxnEvent.win:time(60 sec) group by account.id
For example, assume your application declares the following two types:
create objectarray schema SuperType (p0 string)
create objectarray schema SubType (p1 string) inherits SuperType
epRuntime.sendEvent(new Object[] {"p0_value", "p1_value"}, "SubType"); epRuntime.sendEvent(new Object[] {"p0_value"}, "SuperType");
String[] propertyNamesUpdField = {"name", "addressLine1", "history"}; Object[] propertyTypesUpdField = {String.class, String.class, UpdateHistory.class}; epService.getEPAdministrator().getConfiguration(). addEventType("UpdatedFieldType", propertyNamesUpdField, propertyTypesUpdField); String[] propertyNamesAccountUpdate = {"accountId", "fields"}; Object[] propertyTypesAccountUpdate = {long.class, "UpdatedFieldType"}; epService.getEPAdministrator().getConfiguration(). addEventType("AccountUpdate", propertyNamesAccountUpdate, propertyTypesAccountUpdate);
The next code snippet populates a sample event and sends the event into the engine:
Object[] updatedField = {"Joe Doe", "40 Popular Street", new UpdateHistory()}; Object[] accountUpdate = {10009901, updatedField}; epService.getEPRuntime().sendEvent(accountUpdate, "AccountUpdate");
Last, a sample query to interrogate AccountUpdate
events is as follows:
select accountId, fields.name, fields.addressLine1, fields.history.lastUpdate from AccountUpdate
String[] propertyNames = {"userids", "salesPersons", "items"}; Object[] propertyTypes = {int[].class, SalesPerson[].class, "OrderItem[]"); epService.getEPAdministrator().getConfiguration(). addEventType("SaleEvent", propertyNames, propertyTypes);
The three properties that the above example declares are:
The next EPL statement is a sample query asking for property values held by arrays:
select userids[0], salesPersons[1].name, items[1], items[1].price.amount from SaleEvent
Events can be represented as org.w3c.dom.Node
instances and send into the engine via the sendEvent
method on EPRuntime
or via EventSender
. Please note that configuration is required so the event type name and root element name is known. See Chapter 16, Configuration.
If a XML schema document (XSD file) can be made available as part of the configuration, then Esper can read the schema and appropriately present event type metadata and validate statements that use the event type and its properties. See Section 2.8.1, “Schema-Provided XML Events”.
When no XML schema document is provided, XML events can still be queried, however the return type and return values of property expressions are string-only and no event type metadata is available other than for explicitly configured properties. See Section 2.8.2, “No-Schema-Provided XML Events”.
In all cases Esper allows you to configure explicit XPath expressions as event properties. You can specify arbitrary XPath functions or expressions and provide a property name and type by which result values will be available for use in EPL statements. See Section 2.8.3, “Explicitly-Configured Properties”.
Nested, mapped and indexed event properties are also supported in expressions against org.w3c.dom.Node
events. Thus XML trees can conveniently be interrogated via the property expression syntax.
Only one event type per root element name may be configured. The engine recognizes each event by its root element name or you may use EventSender
to send events.
This section uses the following XML document as an example:
<?xml version="1.0" encoding="UTF-8"?> <Sensor xmlns="SensorSchema"> <ID>urn:epc:1:4.16.36</ID> <Observation Command="READ_PALLET_TAGS_ONLY"> <ID>00000001</ID> <Tag> <ID>urn:epc:1:2.24.400</ID> </Tag> <Tag> <ID>urn:epc:1:2.24.401</ID> </Tag> </Observation> </Sensor>
The schema for the example is:
<?xml version="1.0" encoding="UTF-8"?> <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"> <xs:element name="Sensor"> <xs:complexType> <xs:sequence> <xs:element name="ID" type="xs:string"/> <xs:element ref="Observation" /> </xs:sequence> </xs:complexType> </xs:element> <xs:element name="Observation"> <xs:complexType> <xs:sequence> <xs:element name="ID" type="xs:string"/> <xs:element ref="Tag" maxOccurs="unbounded" /> </xs:sequence> <xs:attribute name="Command" type="xs:string" use="required" /> </xs:complexType> </xs:element> <xs:element name="Tag"> <xs:complexType> <xs:sequence> <xs:element name="ID" type="xs:string"/> </xs:sequence> </xs:complexType> </xs:element> </xs:schema>
URL schemaURL = this.getClass().getClassLoader().getResource("sensor.xsd");
Here is a sample use of the runtime configuration API, please see Chapter 16, Configuration for further examples.
epService = EPServiceProviderManager.getDefaultProvider(); ConfigurationEventTypeXMLDOM sensorcfg = new ConfigurationEventTypeXMLDOM(); sensorcfg.setRootElementName("Sensor"); sensorcfg.setSchemaResource(schemaURL.toString()); epService.getEPAdministrator().getConfiguration() .addEventType("SensorEvent", sensorcfg);
You must provide a root element name. This name is used to look up the event type for the sendEvent(org.w3c.Node node)
method. An EventSender
is a useful alternative method
for sending events if the type lookup based on the root or document element name is not desired.
After adding the event type, you may create statements and send events. Next is a sample statement:
select ID, Observation.Command, Observation.ID, Observation.Tag[0].ID, Observation.Tag[1].ID from SensorEvent
As you can see from the example above, property expressions can query property values held in the XML document's elements and attributes.
There are multiple ways to obtain a XML DOM document instance from a XML string. The next code snippet shows how to obtain a XML DOM org.w3c.Document
instance:
InputSource source = new InputSource(new StringReader(xml)); DocumentBuilderFactory builderFactory = DocumentBuilderFactory.newInstance(); builderFactory.setNamespaceAware(true); Document doc = builderFactory.newDocumentBuilder().parse(source);
Send the org.w3c.Node
or Document
object into the engine for processing:
epService.getEPRuntime().sendEvent(doc);
Consider the next XML document:
<item> <book sku="8800090"> <author>Isaac Asimov</author> <author>Robert A Heinlein</author> </book> </item>
For example, consider the next query:
insert into ObservationStream select ID, Observation from SensorEvent
A further statement can use this stream to query:
select Observation.Command, Observation.Tag[0].ID from ObservationStream
insert into TagListStream select ID as sensorId, Observation.* from SensorEvent
Next is a sample statement to query the new stream:
select sensorId, Command, Tag[0].ID from TagListStream
Please note the following limitations:
This brief example sends an event via EventSender
:
EventSender sender = epRuntime.getEventSender("SensorEvent"); sender.sendEvent(node);
When asked for property names or property metadata, a no-schema type returns empty array.
In all other aspects the type behaves the same as the schema-provided type described earlier.
epService = EPServiceProviderManager.getDefaultProvider(); ConfigurationEventTypeXMLDOM sensorcfg = new ConfigurationEventTypeXMLDOM(); sensorcfg.addXPathProperty("countTags", "count(/ss:Sensor/ss:Observation/ss:Tag)", XPathConstants.NUMBER); sensorcfg.addNamespacePrefix("ss", "SensorSchema"); sensorcfg.setRootElementName("Sensor"); epService.getEPAdministrator().getConfiguration() .addEventType("SensorEvent", sensorcfg);
The countTags
property is now available for querying:
select countTags from SensorEvent
sensorcfg.addXPathProperty("countTags", "count(/ss:Sensor/ss:Observation/ss:Tag)", XPathConstants.NUMBER, "int");
sensorcfg.addXPathProperty("idarray", "//ss:Tag/ss:ID", XPathConstants.NODESET, "String[]");
This code snippet adds two explicit properties and assigns an event type name for each property:
sensorcfg.addXPathPropertyFragment("tagOne", "//ss:Tag[position() = 1]", XPathConstants.NODE, "TagEvent"); sensorcfg.addXPathPropertyFragment("tagArray", "//ss:Tag", XPathConstants.NODESET, "TagEvent");
ConfigurationEventTypeXMLDOM tagcfg = new ConfigurationEventTypeXMLDOM(); tagcfg.setRootElementName("//Tag"); tagcfg.setSchemaResource(schemaURL); epAdministrator.getConfiguration() .addEventType("TagEvent", tagcfg);
The tagOne
and tagArray
properties are now ready for selection and transposing to further streams:
insert into TagOneStream select tagOne.* from SensorEvent
select ID from TagOneStream
An example with indexed properties is shown next:
insert into TagArrayStream select tagArray as mytags from SensorEvent
select mytags[0].ID from TagArrayStream
Part of the extension and plug-in features of Esper is an event representation API. This set of classes allow an application to create new event types and event instances based on information available elsewhere, statically or dynamically at runtime when EPL statements are created. Please see Section 18.8, “Event Type And Event Object” for details.
Creating a plug-in event representation can be useful when your application has existing Java classes that carry event metadata and event property values and your application does not want to (or cannot) extract or transform such event metadata and event data into one of the built-in event representations (POJO Java objects, Map, Object-array or XML DOM).
Further use of a plug-in event representation is to provide a faster or short-cut access path to event data. For example, access to event data stored in a XML format through the Streaming API for XML (StAX) is known to be very efficient. A plug-in event representation can also provide network lookup and dynamic resolution of event type and dynamic sourcing of event instances.
Currently, EsperIO provides the following additional event representations:
Apache Axiom: Streaming API for XML (StAX) implementation
Please see the EsperIO documentation for details on the above.
The chapter on Section 18.8, “Event Type And Event Object” explains how to create your own custom event representation.
To summarize, an event is an immutable record of a past occurrence of an action or state change, and event properties contain useful information about an event.
The length of time an event is of interest to the event processing engine (retention time) depends on your EPL statements, and especially the data window, pattern and output rate limiting clauses of your statements.
During the retention time of an event more information about the event may become available, such as additional properties or changes to existing properties. Esper provides three concepts for handling updates to events.
The first means to handle updating events is the update istream
clause as further described in Section 5.20, “Updating an Insert Stream: the Update IStream Clause”. It is useful when you need to update events as they enter a stream, before events are evaluated by any particular
consuming statement to that stream.
The second means to update events is the on-merge
and on-update
clauses, for use with tables and named windows only, as further described in Section 6.8, “Triggered Upsert using the On-Merge Clause” and Section 6.6, “Updating Data: the On Update clause”. On-merge
is similar to the SQL merge
clause and provides what is known as an "Upsert" operation: Update existing events or if no existing event(s) are found then insert a new event, all in one atomic operation provided by a single EPL statement.
On-update can be used to update individual properties of rows held in a table or named window.
The third means to handle updating events is the revision event types, for use with named windows only, as further described in Section 6.11, “Versioning and Revision Event Type Use with Named Windows”. With revision event types one can declare, via configuration only, multiple different event types and then have the engine present a merged event type that contains a superset of properties of all merged types, and have the engine merge events as they arrive without additional EPL statements.
Note that patterns do not reflect changes to past events. For the temporal nature of patterns, any changes to events that were observed in the past do not reflect upon current pattern state.
Your application events may consist of fairly comprehensive, coarse-grained structures or documents. For example in business-to-business integration scenarios, XML documents or other event objects can be rich deeply-nested graphs of event properties.
To extract information from a coarse-grained event or to perform bulk operations on the rows of the property graph in an event, Esper provides a convenient syntax:
When specifying a filter expression in a pattern or in a select
clause, it may contain a contained-event selection syntax, as further described in Section 5.19, “Contained-Event Selection”.
The insert into
clause can populate instantiate new instances of Java object events, java.util.Map
events and Object[]
(object array) events directly from the results of select
clause expressions and populate such instances. Simply use the event type name as the stream name in the insert into
clause as described in Section 5.10, “Merging Streams and Continuous Insertion: the Insert Into Clause”.
If instead you have an existing instance of a Java object returned by an expression, such as a single-row function or static method invocation for example, you can transpose that expression result object to a stream. This is described further in Section 5.10.7, “Transposing an Expression Result” and Section 10.4, “Select-Clause transpose Function”.
The column names specified in the select
and insert into
clause must match available writable properties in the event object to be populated (the target event type). The expression result types of any expressions in the select
clause must also be compatible with the property types of the target event type.
If populating a POJO-based event type and the class provides a matching constructor, the expression result types of expressions in the select
clause must be compatible with the constructor parameters in the order listed by the constructor. The insert into
clause column names are not relevant in this case.
Consider the following example statement:
insert into com.mycompany.NewEmployeeEvent select fname as firstName, lname as lastName from HRSystemEvent
The above example specifies the fully-qualified class name of NewEmployeeEvent
. The engine instantianes NewEmployeeEvent
for each result row and populates the firstName
and lastName
properties of each instance from the result of select
clause expressions. The HRSystemEvent
in the example is assumed to have lname
and fname
properties, and either setter-methods and a default constructor, or a matching constructor.
Note how the example uses the as
-keyword to assign column names that match the property names of the NewEmployeeEvent
target event. If the property names of the source and target events are the same, the as
-keyword is not required.
The next example is an alternate form and specifies property names within the insert into
clause instead. The example also assumes that NewEmployeeEvent
has been defined or imported via configuration since it does not specify the event class package name:
insert into NewEmployeeEvent(firstName, lastName) select fname, lname from HRSystemEvent
Finally, this example populates HRSystemEvent
events. The example populates the value of a type
property where the event has the value 'NEW' and populates a new event object with the value 'HIRED', copying the fname
and lname
property values to the new event object:
insert into HRSystemEvent select fname, lname, 'HIRED' as type from HRSystemEvent(type='NEW')
The matching of the select
or insert into
-clause column names to target event type's property names is case-sensitive. It is allowed to only populate a subset of all available columns in the target event type. Wildcard (*
) is also allowed and copies all fields of the events or multiple events in a join.
For Java object events, your event class must provide setter-methods according to JavaBean conventions or, alternatively, a matching constructor. If the event class provides setter methods the class should also provide a default constructor taking no parameters. If the event class provides a matching constructor there is no need for setter-methods. If your event class does not have a default constructor and setter methods, or a matching constructor, your application may configure a factory method via ConfigurationEventTypeLegacy
. If your event class does not have a default constructor and there is no factory method provided, the engine uses in connection with the Oracle JVM the sun.reflect.ReflectionFactory
, noting that in this case member variables do not get initialized to assigned defaults.
The engine follows Java standards in terms of widening, performing widening automatically in cases where widening type conversion is allowed without loss of precision, for both boxed and primitive types and including BigInteger and BigDecimal.
When inserting array-typed properties into a Java, Map-type or Object-array underlying event the event definition should declare the target property as an array.
Please note the following limitations:
Event types that utilize XML
org.w3c.dom.Node
underlying event objects cannot be target of aninsert into
clause.
As outlined in Chapter 15, API Reference the interface for listeners is com.espertech.esper.client.UpdateListener
. Implementations must provide a single update
method that the engine invokes when results become available: