Some lower values can occur, cause the size of the tweets differ according to the meta-data contained in the tweet object. We can derive that BaseX scales very well and can keep up with the incoming amount of tweets in the stream. The test show the time BaseX needs to insert large amounts of real tweets into a database. Using BaseX for storing the Twitter Stream 'profile_image_url': 'http://a0./sticky/default_profile_images/default_profile_0_normal.png', 'profile_background_image_url_https': 'https:///images/themes/theme1/bg.png', 'profile_image_url_https': 'https:///sticky/default_profile_images/default_profile_0_normal.png', 'profile_sidebar_border_color': 'C0DEED', 'text': 'Using BaseX for storing the Twitter Stream', It is the pure public live stream without any filtering applied. The following section shows the amount of data, that is delivered by the Twitter Streaming API to the connected endpoints with the 10% gardenhose access per hour on the 6th of the months February, March, April and May. Twitter’s Streaming DataĮach tweet object in the data stream contains the tweet message itself and over 60 data fields (for further information see the fields description). For storing the tweets including the meta-data, we use the standard insert function of XQuery Update.

In the examples section both versions are shown ( tweet as JSON and tweet as XML). For this purpose the parse function of the XQuery JSON Module is used. As Twitter delivers the tweets as JSON objects the objects has to be converted into XML fragments. BaseX as Twitter Storageįor retrieving the Twitter stream we connect with the Streaming API to the endpoint of Twitter and receive a never ending tweet stream. Twitter provides the developer community with a set of APIs for retrieving the data about its users and their communication, including the Streaming API for data-intensive applications, the Search API for querying and filtering the messaging content, and the REST API for accessing the core primitives of the Twitter platform. We illustrate some statistics about the Twitter data and the performance of BaseX.Īs Twitter attracts more and more users (over 140 million active users in 2012) and is generating large amounts of data (over 340 millions of short messages ('tweets') daily), it became a really exciting data source for all kind of analytics. It is about the usage of BaseX for processing and storing the live data stream of Twitter. This article is part of the Advanced User's Guide.

Insert node basex