Open data challenge Telecom Italia

At the beginning of 2014, Telecom Italia launched the first edition of the Big Data Challenge, a contest designed to stimulate the creation and development of innovative technological ideas in the Big Data field. You can find information about it in the following link Open data challenge link Open data challenge link

How to collect location social data from Open Big Data Challenge dandelion API

One of the most interesting datasets in this data challenge is also the Twitter dataset also called Social pulse Milano. It is possible to have this dataset by querying the Dandelion API where Dandelion is the company owner of the data. The data is anonymized and there are no user names or whatsoever privacy sensitive information. However this dataset is pretty interesting as it has a big deal of information. About 270000 geo-referenced tweets from the area of Milan. This dataset can be visualized in order to show social activity from the different areas of the city during the day or the night as you may find in this other . KML visualization guide

In this short guide we will buit a Java client to query the Dandelion API. First thing to do is to make an account at Dandelion API. Once you have the account you'll be assigned a API key and app_id to query their API. This information can be found in the dashboard once that you log in.

The java code that follows is pretty straitfroward : there is the query() method as the key component. This method takes as arguments the API key and secret, the dataset name you find in Dandelion, the location = Milano and the limit and offset parameters for the query itself. The method returns an JSONObject which is a string of text that has many fields. We parse these fields one by one in the following lines.

There is only just a snippet of code for explanation. At the end of the page there is the link for project download


     public class Main2SocialPulse {
	
	public static void main(String[] args) throws IOException {

		String app_id ="app_id";  // API secret and key
		String app_key ="API key";
		String dataset = "social-pulse-milano";
		String where = "";
		int limit = 10;
		int offset = 0;

		JSONObject response1 = query(app_id, app_key, dataset, where, limit, offset); // method for querying the dandelion API it takes as arguments
        //the API key and secret, the dataset name you find in Dandelio, the location = Milano and the limit and offset parameters for the query itself 
		//the method returns an JSONObject which is a string of text that has many fields. We parse these fields one by one in the following lines
		int count = response1.getInt("count");
		System.out.println("TOTAL COUNT = "+count);

		FileWriter fw = new FileWriter(new File(""+dataset+".csv"));
		String sep ="\t\t\t";//delimiter
		String headers ="user entities language municipality_name date timestamp coords";

		String[] headers_array = headers.split(" ");
		
		

Important note: the code has to import some classes from other libraries. So it is better to download the entire project from here :

By running this code you can start downloading the SocialPulse dataset in a csv file.

There is an interesting example of how to visualize this dataset in KML KML visualization guide

If you enjoyed this article guide please share

Tweet

or follow ..

Follow @datamap2t