Aggregated data, as defined by Wikitonary:
The state of being collected into a mass, assemblage, or sum
In web terms, this means the sum of data, as a group, taken from various feeds and be useful for later output to provide a wholesome outlook of the participant particulars.
An example of using aggregated data is taking output from a particular subject from Twitter and Sky News and grouping the data to output the latest information of what’s occurin’
Here are five tips towards creating a decent aggregated feed based on various sources. Although they may not be perfect or useful in all situations, they should provide a basic outline on what you should be doing when setting up your feed.
Tip #1
Make sure your sources are reliable! Although in some cases; i.e. taking data from social reports such as Twitter, where information given may not be 100% truthful – it may be hard/impossible to filter out the useless “WHAT’S #MAJORHAPPENING ABOUT TWEEPS?” tweets, they’d still provide an increment to how important the event may be.
Collecting your data from notably reliable sources such as:
- News websites such as BBC, Sky, ITV, Guardian etc
- Citizen reporters who have a reputation for providing reliable reports.
- Social media websites which are known for reporting true facts.
Should provide you a higher chance of providing your users with true and up to date information.
Tip #2
Stay ahead of others! Try to collect data from realtime sources such as:
- Website feeds
Tip #3
Prioritize your sources! Although I could go into how you should have algorithms setup to categorize and prioritize the information the site that it’s currently processing, people shouldn’t expect normal people to be able to come up with something like that on their own in the time of a potential crisis or several worldwide catastrophes.
Tip #4
Use feedback from your visitors! If your site is providing your users with out of date or incorrect data, then it could cause problems and confusions down the line. When this happens, you’ll start to lose visitors and potential revenue. Provide a way for your users to send you feedback on the data you’re displaying and make appropriate changes when something isn’t working as it should.
Tip #5
Know your facts! Back when China had a massive earthquake in 2008, Twitter was the first notable source to the incidents happening, but what tops the triumph off even more is the fact that the users beat the USGS with releasing the news by a pretty staggering 3 minutes. – Source and stuff.
This proves how much trusting your sources can help and knowing that Twitter was the first to release the news, means that you can pretty much guarantee that it’ll happen again and again, the more the user base grows.
This post is a bit different to my others since I’m not really writing about the happenings and what-not, it’s more of a public notepad for me to come up with ideas for my application; iTunesTweet.
The ideas that I write about may or may not become a part of the stable release of iTunesTweet. This could be due to various reasons:
- It’s not stable, i.e. it doesn’t work in the way that it was intended to or it has too many bugs for me to take care of at one time.
- There is no need for the feature. Some times I may come up with an idea that I think would be useful, however in real world testing, the majority of users would think otherwise.
- It changes a big part of the code. I don’t have time to re-write the main system again, so features will be currently limited to the size and time they’ll take to integrate.
So now that I’ve covered my ass from people who may get annoyed that a feature they really like isn’t in the application, I can begin writing ideas!
Now playing feed
This would be a global feature available on the website (once the re-design is finished). With a few modifications to the log sequence it would be possible for me to see how many songs are currently playing (ignoring the fact the user may press pause) and display them in a feed on the website.
This shouldn’t be too hard to integrate since I’d only have to work out when the song will stop playing (TTL for the song).
Random maths jibber
Current time – (song length + length played for) : something like that anyway
I think this feature would open up many new possibilities for the application. A feed of currently playing songs across the world would be a brilliant way to promote music. The more people playing the song, the more it stands out.
We could even take into consideration the rating the song has, and use this to promote it further.
Share playlist
I promise this has nothing to do with the recent movements by Spotify which now allow you to share playlists to friends at Twitter and Facebook!
Technically it should be possible for me to find out the playlist in which the currently playing song resides in. Through this it would then be possible for the user to share that playlist on their Twitter account. The website would also be able to group playlists by Twitter user (which could cause problems if the user changed their username, but it’s a risk).
It’s not a big idea nor particularly complicated but it’s a nice way to share music with the world.
Better error handling
This is a given, something that I should have implemented a while ago. Due to Twitters recent surge in problems, iTunesTweet needs to be equipped with enough decisions in its code to handle as many problems as I find.
Thanks to Cohen I was able to fix one problem this morning. And since Twitter fails so much at the moment, now is a brilliant time for me to teach iTunesTweet to handle all kinds of errors.
Eventually the error log file will consist of a deeper analysis of the problems including:
- Epoch/unix time stamps
- All data that iTunesTweet was sending
- Twitter status code
- And whatever else I deem useful to further investigation
The resulting file will automatically be sent to the server when it reaches a preset amount of errors, or you can choose to send it if the error level is bugging you.
Song correction tool
We all have songs in our libraries which have imported names such as “03 David Guetta – Memories” or with spelling mistakes “Dvd Guetta – Memorys” – maybe it’s just me, but I find them annoying and I hate correcting them as I find them.
Wouldn’t it be awesome if iTunesTweet could analyze every song in your library and automatically correct or suggest the real spellings?
I’ve applied for an API key at Song.ly in the hope to integrate this feature into the code! To reduce the amount of API calls iTunesTweet makes, it would be a good idea for my server to cache all of the original and correct strings and then run the files against the server. Once I have a base library of a few thousand songs, I could further the development of the tool to check my server before song.ly – see I play nice with API developers.
Album art for those which iTunes Store can’t find
Some times iTunes Store can’t find album art to a song because it’s based on both the artist and the song title or because iTunes just fails at these kinds of things.
It should be possible for me to write a custom album artwork handler for all of the songs that iTunes can’t handle.
Keep in mind that this feature would still be limited to how accurate the song information is, although a mix of this and the song correction tool should give you good results most of the time!
More to come
Some more ideas will probably arise because of these, or betas will come out with me testing them, so keep an eye out 
Previously I discussed oAuth via desktop applications, specifically why I chose to use it in iTunesTweet and a few of the benefits it has over the normal API. In this post I’ll be comparing oAuth over xAuth, explaining the benefits and problems each standard face.
What is oAuth?
oAuth stands for Open Authorization. It’s an open standard which allows users to share data between their system and a server without having to enter their username or password to the 3rd party website. This means that no personal data is stored by the website/application, only tokens which allow the site to contact the service for basic details.
These tokens work only for the user they’re generated by and the specific services and requested resources. - Source.
What is xAuth?
xAuth is a crossover between oAuth and basic API. It takes the basic functionality of oAuth and provides developers with the same token system, but allows them to use the API at the same time.
It’s actually a little bit weird since xAuth doesn’t use any process to request tokens, it simply takes the users credentials, allows you to make an API call and receives the tokens back in return.
Wait, so xAuth takes user credentials, how is this safe?
And this is the downside of xAuth. It requires both user and password in return for access to the API. At first it sounds like you might as well just use the API as it is, and you’re right, you probably should for simplicity, but xAuth provides some what decent security with the full API power.
It’s only safe because it stores the credentials, and even then to relate it to oAuth (known for security) is in my eyes… stupid.
How does it work?
Firstly, you send an email to Twitter API asking if they’ll register your oAuth account with xAuth (seems weird, but okay…). If you get approved, you then need to call the xAuth API method https://api.twitter.com/oauth/access_token with three parameters:
- x_auth_username – obviously the requesting username
- x_auth_password – obviously the requesting password
- x_auth_mode – this needs to be set to client_auth
When you get the tokens back, you should store them as they don’t expire. They can then be used for the API calls, the password can then be dropped. So I guess it has a bit of security.
Just so long as you keep in mind that xAuth is one API call, with no request tokens used, you’ll be fine. Keep in this mind and you’ll be fine!
I actually managed to get a second version of iTunesTweet (me testing) using xAuth in a couple of hours – I had to re-write some of the code to handle the outputs.
I had a long weekend to myself this week and thought that I should use it wisely. Probably preparing for my new job, but nope, I decided to finally recode iTunesTweet. For those readers who have no idea what I’m talking about, check this post I made.
The biggest improvement to the re-write is the oAuth implementation, which of course is the subject of this post. Why? Well because I wanted to discuss the different ways that the technology can be used (in regards to connecting with Twitter) and why I chose one over the other.
I also discussed this topic at Stack Overflow and asked for professional opinions on the subject. They completely agreed with the methods I shall now talk about. If you want to see the topic, click here.
What is oAuth?
The first question to such a post is; what is oAuth? What is this awesome new technology that websites such as Twitter, LinkedIn and Google are using?
oAuth stands for Open Authorization. It’s an open standard which allows users to share data between their system and a server without having to enter their username or password to the 3rd party website. This means that no personal data is stored by the website/application, only tokens which allow the site to contact the service for basic details.
These tokens work only for the user they’re generated by and the specific services and requested resources.
Why does iTunesTweet use oAuth?
In Version 1, iTunesTweet stored both username and password in the registry, so that it could call Twitter’s REST API to update status, follow iTunesTweet et cetera. Although encrypted, it’s still open to vulnerabilities. And that’s something we don’t want, right!
iTunesTweet V2 now uses oAuth to secure data being sent and received between itself and Twitter.
If you are so concerned about users security, why did it take so long?
A very good question if I may so myself! The problem with oAuth is that it uses some extremely complex data processes, which is irritably hard to implement into a scripting language such as that, of which iTunesTweet is created in.
At the same time of wanting to keep users secure, I did not want to provide them with a crap implementation of something which should be securing their data. I’m not going to release something I know comes with a bogus claim.
Why not stick with basic REST API calls?
Not only do they pose as a security threat but Twitter is shutting down the API too. The announcement can be read here. For those awaiting the day, check out this awesome Twitter API Countdown death clock.
So you’re using the application interface for Twitter’s oAuth security?
Simple answer? No. Extremely boring and possibly long answer?
As I said previously, it is irritably hard to implement oAuth into the language I’m using. This alone would give a lot of developers the notion to swap to a capable language, heck, I even started to myself using C#.
The reason that I decided to leave the new implementation behind and go back to my original base was that I realized that iTunesTweet is literally an addon to iTunes. Although not an official one, it does access iTunes API and as such, adds functionality that is missing. After realizing this, I wanted to keep iTunesTweet small, like addons should be. Although AutoIt is relatively slow compared to C# for example, it also gives me a much better base set of functions which make it easier to hook into the API. Not only that, but with some special compilation rules and packers I can keep the size down, even with the 6 (at current count) resources it includes.
So you used the web access for a desktop application? Correct. If you’re asking why or not, you’re going to get the answer. It’s simply because the web authorization method is much easier to implement. Thanks to the creators of EpiTwitter/EpioAuth, I’m able to host the web files and make secure connections to them between my application and the files.
The application interface requires us to store a number and post it back and do this and that and reverse and encode and blah blah blah. It’s long and difficult. Why beat around the bush? I can simply send the users secure tokens to the web application that I host, and Twitter does the rest. Even if the data is hacked, they can’t do anything with it, since the files I host do all the formatting work anyway.
So you’re using a web interface for a desktop application?
Exactly! A lot of applications do it too, why should iTunesTweet be any different?
How do you get the users to authorize the application then?
Simple, I provide a good (and I use that term lightly since it’s going to be changed for the final version) little interface in the form of an IE window (without the Chrome etc) and ask them to click a link. The link takes them to the Twitter page, they enter their details, it re-directs back to my site and I ask them to close the window.
Do you collect any data during this process?
Yes, a couple of things are collected, which later make iTunesTweet run:
- Twitter oAuth token
- oAuth token from username
- oAuth secret token from password
- Twitter username (this is deleted as soon as the iTunesTweet account is followed)
- Twitter picture – this is only during beta, it was originally going to be used for updating the users status themselves
And that’s it. Nothing harmful, usernames are publically available anyway, and since it’s deleted within 2 minutes, it causes no harm.
So there.
I proved absolutely nothing except that in some circumstances it is better to use tools available to you because it makes your job easier and provides the same functionality just in a different model. Perhaps we’ll see some other desktop applications which use web resources like iTunesTweet does?
At the end of January I released a program I’d been working on for a while, iTunesTweet. If you can’t guess what it does from the name, then take a step back and just think about it 
The application simply tweets what you’re currently listening to on iTunes, to Twitter.
3 months later and iTunesTweet has been downloaded over 100 times and over 6000 songs have been logged – iTunesTweet optionally allows you to send data to my site, otherwise I know nothing! I’m personally very proud on the statistics iTT has stacked up! It’s quite resounding to know how much music is played, and I can’t thank everyone who has opted in to the data collection scheme enough!
Some of the statistics were erased when I changed the way storage works, unfortunately I was unable to convert previously stored data because of the routines the data goes through. I also provide a little page which you can see how many songs are played per day and the previous days too! Check out iTunesTweet Usage Graph.
Recently I’ve been working on a number of fixes that have caused the application to respond incorrectly to certain iTunes events – specifically when iTunes tries to close. And of course, new features!
I hope the new features I’ve been implementing make it easier to use and make iTT a better application over all. So what exactly are these features and how will they improve iTT? Currently to change any settings, unless you know how to use Regedit, require you to practically re-install iTunesTweet which is not good – I know – so I’ve been working on a nice “Options” window which lets you change settings quickly and easily! The options GUI is a completely custom drawn window using some awesome GDI. By adding this new window, I’m able to add new functions and their available options with no problems!
As of now, iTunesTweet notifies you of the song your playing through a tray tip, which is kind of boring.
Boring!
Now however, I’ve added a toast! Similar to Windows Live Messenger, iTunesTweet now notifies you in the same manner only using a custom toast popup. I still have some work left to do it, the boring grey probably wont stay for too long.
I’ve also changed the way it contacts my website for usage details. It now requires the iTunesTweet user-agent which is a stop against spam agents messing statistics (not that they have, but I’m prepared).
A new update function automatically updates iTunesTweet when there is an available download. This is a feature I love in every application and makes using the program problemless!
In the future, I intend to add scrobble support – only if Last.FM isn’t doing so already. This means that not only will Twitter know what you’re listening to, but Last.FM too! Two networks in one!
Recent Comments