Final Project #3 – Tutorial

I started my work on the final project by following a tutorial – FUN with FACEBOOK in NEO4J wrote by Rik Van Bruggen. I followed the article as well as video. As I have quickly realised, the author took some of the steps for granted and didn’t include them in the tutorial.

In this post I’ll re-write it and make sure that anyone (even less technical people) can follow the simple instructions.

On top of that, I’ll have to update a few things, as the old method doesn’t work with the newest release of the Neo4j.

Step 1 – Give me my data

The first step is to get data from Facebook. I’ve used the tool Give Me My Data (described in more details in the previous post), but if you’re a software developer, you can use Facebook API directly.

After downloading the tool and linking it to your Facebook account, you have to download a particular set of data, which in our case is:

Data: Mutual Friends Network Graph

Format: CSV

It should look like this:

Untitled

 

Step 2 – Copy and paste results into a text file

I have downloaded a tool used by Rik in his tutorial – Sublime Text 2. This is the output:

Screen Shot 2014-12-14 at 11.29.23

Now, save the file, MAKING SURE it’s a .txt file (so for example name if: FacebookData.txt, and click SAVE).

Step 3 – Uploading data into Google spreadsheets

Now that your .txt file is ready, you need to upload it into Google Spreadsheet.

Here are all the steps:

– Click on File –> Import…

– Upload the FacebookData.txt you’ve just created, choosing the following settings:

Untitled

 

And is the final output:

Untitled1

Step 4 – Preparing data for creating cyphers

In order to upload data into Neo4j, we need to convert it into cyphers.

The first formula will eliminate all the dupes. Name your column D “No dupes in uid1 or uid2”, and use this formula:

=UNIQUE(A2:A5240)

*the second value (A5240) should be the last cell of your column A and it’ll depend on how much data/Facebook friends you have.

The second formula in column E will “Combine uid1 and uid2”, and it looks like this:

=UNIQUE(D2:D654)

*the second value (D654) should be the last cell of your column D and it’ll depend on how much data/Facebook friends you have.

Step 5 –  Creating cyphers

Now we’re ready to create the cyphers. After struggling with uploading data into Neo4j for a few days, I have realised the old cyphers (from the original tutorial) are outdated and won’t work…

Fortunately, finally I have found a solution to the problem, and so now I can give you the RIGHT formulas.

5.1. Cypher statements for nodes

Use column G and the following formula:

="create (n:Facebook {name:'"&E2&"'});" 

Then paste it into the whole column.

5.2. Cypher statements for relationships

Use column I and the following formula:

="match (n1:Facebook {name:'"&A2&"'}),(n2:Facebook {name:'"&B2&"'}) create n1-[:IS_A_FRIEND_OF]->n2;"

Then paste it into the whole column.

This is the output you should have (ignore columns F and H and leave them empty):

Screen Shot 2014-12-14 at 12.24.48

 

Step 6 – Downloading and installing Java and Neo4j

Before you download Neo4j, you need to make sure that you have Java installed on your computer. If not (as it was in my case), you can download it from here. They have all the instructions, but it’s not complicated.

When it comes to installing Neo4j, it turns out to be NOT so easy if you’ve never done a similar thing before. I struggled fo a bit, but then found a very useful YouTube tutorial – Install Neo4j Mac or Unix. If you’re using Windows, take a look at Install Neo4j Windows instead.

Step 7 – Creating the index for property name based on label Facebook:

Before you begin adding data, open Neo4j and go to Console (shell environment), which looks like this:

Screen Shot 2014-12-14 at 12.31.29

And type in:

CREATE INDEX ON :Facebook(name)

It’ll create the index for property “name” based on label “Facebook”. Without this step, the next ones won’t work.

Step 8 – Uploading cyphers into Neo4j

You can upload your cyphers either into web based Neo4j console (the ones used before) or your own computer console (on Mac it’s called Terminal).

Using terminal may work better, but first you need to remember to open the shell-like environment using a command:

bin/neo4j-shell

It should look like this:

Screen Shot 2014-12-14 at 12.57.41

Now, all you need to do (in theory) is wrap your cypher statements for node into “begin” and “commit” (to start end end the transaction), and paste them into one of the consoles.

Unfortunately, at least in my case, neither of the consoles would read cyphers as separate transactions, and so I had had to add “begin” and “commit” before EVERY SINGLE CYPHER. Fortunately you don’t have to do it manually.

If you look back here…

Screen Shot 2014-12-14 at 12.24.48

… you can see that I added columns F with “begin” and H with “commit”. Copy and paste all three columns into a Word document.

(Paste special –> Unformatted text will get read of the table)

 

 

 

You’ll get this result:

Screen Shot 2014-12-14 at 13.05.27

But what we need is:

Screen Shot 2014-12-14 at 13.07.41

To move “begin” and “commit” into separate lines, you can use a Word “Replace” formula (see where to find it below) and replace some of the text with a “Manual Line Break”, which will automatically insert new lines with text you specify (e.g. “begin”).

Screen Shot 2014-12-14 at 13.08.52

 

When ready, paste everything into the console, and it’ll upload all your nodes.

Prepare the “Cypher statements for relationships” in the same way (adding “begin” and “commit”) and uploaded them into Neo4j as well.

In the console, it’ll look like this:

8b-g2KUllSRvgp9B3B2-5sXxQ4nd5xnKPUSxMJfxCG-_MroWXK1Do2LwbA_4lKMX_EiNg1DGwMwDEyZVKseWQ4DlvCUGokCIBcnLie9UKxLuoYrmQJC6zuN8A

(*This photo is taken fro the original tutorial, and so your cyphers inside should look differently)

Now, you should see both nodes and relationship ready, like here:

Screen Shot 2014-12-14 at 13.15.22

 

Step 9 –  Exploring your data

The last step is to go to “Data browser” (in Neo4j) and type:

match (n) return (n)

It’ll give you a list of all the nodes like this:

Screen Shot 2014-12-14 at 13.29.51

 

Now, open any of them, and click on the “Switch view mode” (top right corner):

Screen Shot 2014-12-14 at 13.30.52

 

You should now, finally, see your graph:

Screen Shot 2014-12-14 at 13.32.15

 

 

Step 10 – Start exploring your graph database!

That’s it, you’ve uploaded data into Neo4j and it’s ready for you to explore! You can change the design in the Neo4j Browser (see the tutorial on How to style the Neo4j-Browser visualization?). There’s more (usually more advanced) techniques described here.
Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 International License.

Final Project #2 – About Give Me My Data and Neo4j

Having explored a few different software for data visualisation (described in post #1), finally I came across two tools that combined enabled me to work on my final project.

Give Me My Data

The first one of them, Give Me My Data, helps to reclaim and reuse Facebook data. Connected to Facebook account, it enables users to download a specific set of data (e.g. connections or photos) in a chosen format (e.g. CSV, HML or JSON).

Here are examples of different ways of visualising that data published by Owen Mundy, who developed the software:

Screen Shot 2014-12-13 at 17.11.53

 

Unfortunately, extracted data isn’t enough to create a stunning visualisation. I kept looking for more advanced software for data visualisation, and finally I came across a tutorial that explained how to create an interactive map of Facebook network with the use of Give Me My Data and Neo4j.

Only now do I know that following tutorial does not equal an easy job… What was meant to take exactly 8 minutes and 18 seconds, turned out to take DAYS (weeks in total).

Neo4j

Neo4j is an open-source graph database, implemented in Java. In short, it helps with visualising and exploring data through interactive graphs.

Being inspired by a blog post “FUN with FACEBOOK in NEO4J”, I have decided to go ahead, follow the tutorial (also a video tutorial) and create my own interactive Facebook network, that was meant to look like that:

EDUexhCc7JpnONQsl2Lj66t78t1eYSFhrlC3-rlWB-Xi6Ifm_ZcprjX69ddMQeMD4HGcGx-cqqM7LTffyQvBWXGflm4JunBl9wYs6Erq2nBr_aGU7w8I_OePUw-1024x567

(photo take from the tutorial, link above)

 

As I have found out rather quickly, uploading data into Neo4j WAS NOT easy at all, and required actual programming skills that I didn’t have. Have I known that in the beginning, probably I would have abandoned the idea in the very beginning and focus my final project on something I could actually understand.

Yet before I had realised that the task wouldn’t take 8 minutes, I put so much effort in trying, that I knew I had no choice, but to go ahead.

 


 

In my next blog post I’ll explain why instructions from the tutorial were wrong, and how it took me days to figure out the “right way” of doing things. I’ll try to re-create what I’ve learnt and done, and create my own updated tutorial for newbies like myself.

 

 

 
Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 International License.

Final Project #1 – Exploring tools for visualising data from social networks

Wanting to visualise and map networks, I have started with exploring ways of doing so.

InMaps

First of all, I needed a network, which would became my digital database. I was very excited to discover a tool called InMaps, which enables LinkedIn users to visualise and map their connections.

As described in more details by a  blogger Ingrid Lunden, “The service was first launched in January, 2011 as a way for you to look at different clusters in your network, color code them, and see where they may cross over with each other, with separations happening often around current and previous jobs, education, social circles and so on. Through this you could also interact, by clicking on hubs to see different profiles.” (read the whole article)

Here is an example by Paul Downey:

6249332030_0d6b272fb2_o

Source

Unfortunately, InMaps have been disabled by LinkedIn in September 2014, and so I had too keep looking for different tools.

My next choice of a social network was Facebook as it’s the biggest database I own with my 370 “friends”. In comparison, I follow 278 people/companies on Twitter, but have only 130 followers.

I have started searching for “tools for visualising Facebook network”. Some of them were no longer active, but I’ve managed to use the others.

Connect 

The first app I’ve explored was Connect. It works both on smartphones and web browsers. Linked to Facebook account, it “maps” your Facebook network by displaying your friends on an interactive Google Map. Here are examples of my own map created by using Connect:

Connect

 

Social Map

The other software I’ve used is called Social Map. The software has been developed for the NACIS: Student Web Mapping Competition by Zdenek Hynek and Martin Pulicar from Masaryk University, Czech Republic. It uses Facebook API to

It serves as an “interactive map enabling spatial exploration of personal network of Facebook friends. The project has been initially conceived to find out how deep [one] could dig into people’s spatial relationship using the API of everyone’s favourite social network.” (source)

 

 Here are some screenshot:

Untitled

 

WolframAlpha

The last tool I have been exploring was Wolfram Alpha, and it was far more advanced than the two previous ones. As stated on the website:

“Wolfram|Alpha introduces a fundamentally new way to get knowledge and answers—
not by searching the web, but by doing dynamic computations based on a vast collection of built-in data, algorithms, and methods.” (source)

Yet out of all the features they offer, I was interested only in Personal Analytics for Facebook. In order to generate a full report and Facebook analytics I had to update my account to Pro. Thankfully, student/academic discounts are available.

The full report is available here.

As my main interest lies in my visualisations and mapping, I have focused on two particular parts of the report: Friend Network and Color-coded friend network. By choosing various databases (e.g. “all friends”, “female friends” or “friends from my hometown”), I was able to create various representations of my Facebook network.

Here is the final output:

Untitled

Top left: “All friends”

Top right: “From where I live”

Bottom left: “From hometown”

Bottom right: “Older than me”

Untitled1

Left: “Female”

Right: “Male”

Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 International License.