Working with GA analytics from Python API

To analyse data from your website: users, sessions, page views, many would choose Google Analytics – GA (https://analytics.google.com/). To obtain data you can go to the GA website, choose metrics and time frame you need and download data to work with in csv or xslx format. However, there is also an API to directly import data from GA to data-analysing script (e.g. within Jupyter notebook), so why not explore it instead?

How to connect to GA API and use data from it? Great step-by-step instructions are available on GA API websites (https://developers.google.com/sheets/api/quickstart/python) and I will just combine all of needed info in one place.

First of all:

  1. Go to this page: https://console.developers.google.com/flows/enableapi?apiid=sheets.googleapis.com and create a project and (automatically) turn on the API. Click Continue, then click Go to credentials.
  2. On the Add credentials to your projectpage, click the Cancel
  3. At the top of the page, select the OAuth consent screen Select an Email address, enter a Product name if not already set, and click the Save button.
  4. Select the Credentials tab, click the Create credentials button and select OAuth client ID.
  5. Select the application type Other, enter the name and click the Create.
  6. Dismiss the resulting dialog.
  7. Click the file_download(Download JSON) button to the right of the client ID.
  8. Move this file to your working directory and rename it to client_secrets.json
  9. on your computer install google-api-python-client
    pip install --upgrade google-api-python-client

    or

  10. sudo easy_install --upgrade google-api-python-client

and you’re good to go.

Now how to connect to GA and send your first request:

(the example is based on https://github.com/EchoFUN/GAreader/blob/master/hello_analytics_api_v3.py)

from googleapiclient.errors import HttpError
from googleapiclient import sample_tools
from oauth2client.client import AccessTokenRefreshError

def ga_request(input_dict):
    service, flags = sample_tools.init(
    [], 'analytics', 'v3', __doc__, __file__,
    scope='https://www.googleapis.com/auth/analytics.readonly')

    try:
        first_profile_id = get_first_profile_id(service)
        if not first_profile_id:
            print('Could not find a valid profile for this user.')
        else:
            results = get_top_keywords(service, first_profile_id, input_dict)
            return results

    except TypeError as error:
        print(('There was an error in constructing your query : %s' % error))

    except HttpError as error:
        print(('Arg, there was an API error : %s : %s' %
              (error.resp.status, error._get_reason())))

    except AccessTokenRefreshError:
        print('The credentials have been revoked or expired, please re-run '
          'the application to re-authorize')

def get_first_profile_id(service):
    accounts = service.management().accounts().list().execute()
    if accounts.get('items'):
        firstAccountId = accounts.get('items')[0].get('id')
        webproperties = service.management().webproperties().list(
            accountId=firstAccountId).execute()

        if webproperties.get('items'):
            firstWebpropertyId = webproperties.get('items')[0].get('id')
            profiles = service.management().profiles().list(
                accountId=firstAccountId,
                webPropertyId=firstWebpropertyId).execute()

            if profiles.get('items'):
                return profiles.get('items')[0].get('id')

    return None

def get_top_keywords(service, profile_id, input_dict):
    if input_dict['filters'] == '':
        return service.data().ga().get(
            ids=input_dict['ids'],
            start_date=input_dict['start_date'],
            end_date=input_dict['end_date'],
            metrics=input_dict['metrics'],
            dimensions=input_dict['dimensions']).execute() 
    return service.data().ga().get(
        ids=input_dict['ids'],
        start_date=input_dict['start_date'],
        end_date=input_dict['end_date'], 
        metrics=input_dict['metrics'],
        filters = input_dict['filters'],
        dimensions=input_dict['dimensions']).execute()

Save this file as ga_api_example.py

You need to remember that from API, metrics and other feature names may have other exact names, e.g. custom dimensions are just called dimension[no] etc.

And now, after loading your file

from ga_api_example import ga_request

You can prepare request

request = {
"ids" : "ga:<your_id>",
"start_date" : "2017-06-25",
"end_date" : "2017-06-25",
"metrics" : "ga:pageviews",
 "filters" : "ga:dimension1=~yes",
"dimensions" : ""
}

=~ means use regex as at GA website reports

and execute it:

data = ga_request(request)

Now you have data in Python script where you work with, e.g. with pandas.

Seaborn library for pretty plots

Seaborn is visualization library based on matplotlib (and complementary to matplotlib, you should really understand matplotlib first). It basically makes your work easier and prettier. The library is not really complicated and broad but it makes some thing for you, things that you would have to do in matplotlib on your own.

Seaborn works very well with libraries used in Python for data analysis (pandas, numpy, scipy, statsmodels) and may be used easily in Jupyter notebook for plots imaging. The most frequently mentioned advantage of seaborn are built-in themes. Well… it helps people who don’t know how to combine different colors to make plots aesthetic. Second thing is that functions do really nice plots that try to show something useful even when called with a minimal set of arguments. However, as for matplotlib you have almost endless number of possibilities to adjust your plots.

Additional information about seaborn can be find here: https://seaborn.pydata.org/

Installation is pretty easy:

pip install seaborn

or

conda install seaborn

Additionall info about installation (including development version of seaborn) can be found here: http://seaborn.pydata.org/installing.html

They did really nice job also when comes to documentation and tutorials (e.g. https://seaborn.pydata.org/tutorial.html)

My favorite thing about seaborn? I would say seaborn.distplot function. I usually do two visualizations to look at data before working with it – scatter plot and distribution. Scatter plot is probably easy to obtain in any visualization lib you work with, as is with matplotlib. However to see distribution along with KDE plot, I recommend seaborn function displot.

Here are some examples: https://seaborn.pydata.org/generated/seaborn.distplot.html?highlight=distplot#seaborn.distplot

Basically, all you need to do to have your data in array, e.g. as pandas DataFrame column

import seaborn as sns
ax = sns.distplot(df['column name'])

To try with example data, you can try plotting normally distributed data:

import seaborn as sns, numpy as np
x = np.random.randn(100)
ax = sns.distplot(x)

 

DATA Science in business – perspective from first day employee

This is the one and only opportunity for me to write what was my thoughts about data science after first couple of days in a new office. I decided to start another job, first job in data analysis and first in software house – huge difference from the first day.

First of all I had no idea what anyone was talking about. All those abbreviations and  office slang is a bit overwhelming at first. But you get used to it and understand more and more every day. But after couple of days it is still not enough. But hey! During the first weeks you’re allowed to ask stupid questions, so use it.

Second of all, your chosen technology is not necessary used in the office. Even if you spoke about used technologies during interview and you were asked to implement your chosen technology to your new office, you will have to use also technologies they use. It is not at all a bad thing, it gives you opportunity to expand your knowledge and perspectives. It is easier to understand the data in a way they work with it and after you understand it you can go on with your technologies.

Third thing is the co-operation. You will not be with data alone, others need your results, they (engineers) change the way how data looks like, they want to learn how to do some analyses on their own – strong co-operation in data analysis is crucial.

Forth – you need to understand the goal of your existence in the company. You don’t analyse the data just to analyse it, there is company strategy in it, you have to keep it mind all the time.

Last, but not least, connected to all of the above, company lives its life and what you do today may not be so necessary tomorrow and sometimes you have to deal with unfinished projects. Just go with the flow (and company development).

Get noticed – wrapping up

So the contest I was participating in – get noticed – has ended. It triggered me to start this blog and I think I will continue to write about bioinformatics, data science, Python and so on (however, maybe one post per week will be easier to manage).

I am really happy that I lasted and I am among 183 finalists (from over a 1000 contestants).

I had a really great adventure during this contest. I didn’t finish my project but the end of the contest does not mean the end of the project.

I even got the courage to apply for a job in data science and I got it (It will be my third job but I hope I’ll manage).

In June I plan to start real web server with the modules I managed to develop during the contest. So keep in touch.

Testing modules within cherrypy

I was wondering how unittests may be implemented within cherrypy to test not cherrypy itself but the modules included. I wanted tests to be run every time when new changes were introduced, that means when the server reloads.

There are some information about unittesting in cherrypy (e.g. https://stackoverflow.com/questions/14260101/unittesting-cherrypy-webapp), however, none directly addressed my issue.

I decided to prepare single file named unittests.py (remember not to name your file as unittest.py as there will be problem with library import) with unittests and TextTestRunner setup. TextTestRunner does not interrupt the reload of the server as standard unittest.main() does.

import unittest
from module import function1, function2

class MyTestClass(unittest.TestCase):

    def test_search(self):
        self.assertEqual(function1(input1), output1)
        self.assertEqual(function2(input2), output2)

def start():
    suite = unittest.TestSuite()
    suite.addTest(MyTestClass("test_search"))
    runner = unittest.TextTestRunner()
    runner.run(suite)

and in the main cherrypy file, besides import, I added single line (unittests.start()):

import unittests

if __name__ == "__main__":
    cherrypy.config.update("config.conf")
    unittests.start()
    cherrypy.quickstart(Page(), config="config.conf")

Every time the server reloaded the tests were run and I could check whether everything is correct.

Python in Big Data #1 – Hadoop & Snakebite

One of the often mentioned term during searching for Big Data information is Hadoop. What is Hadoop exactly? And what are first steps to handle Hadoop with Python?

Hadoop is a system file (HDFS) that enable scalable data handling. Basically, Hadoop was developed to store large amount of data while providing reliability and scalability. It is based on data block system. HDFS is based on two processes: NameNode that gathers metadata and DataNodes that store blocks of data. Blocks of data are replicated on different machines to provide data stability when one machine crash.

There are increasing number of libraries for Python that enables handling Hadoop. Just to name some: Snakebite, mrjob, PySpark.

Snakebite enables basic file operations on Hadoop and accessing Hadoop from Python applications. It can be install easily:

pip install snakebite

While installed, within Python application you can connect with HDFS NameNode with:

from snakebite.client import Client
client = Client('localhost', 8000)

Then you can implement various functions similar to functions in shell: ls, mkdir, delete to handle files and directories within Hadoop. Another group of functions (e.g. copyToLocal) enable retrieving data from HDFS.

from snakebite.client import Client
client = Client('localhost', 8000)
for file in client.copyToLocal(['/hdfs_1/file1.txt'], '/tmp'):
    print file

Snakebite provide also CLI client.

Some additional information can be found in a free book:

Hadoop with Python by Zachary Radtka and Donald Miner

http://www.oreilly.com/programming/free/hadoop-with-python.csp

How Python connects you with biological databases? #2 – PubMed

Pubmed is the biggest database of biological scientific papers. There are other databases that gather information about all scientific papers (e.g. google scholar, scopus), however in biological sciences, still most commonly used is Pubmed from NCBI.

There is quite easy way to access Pubmed through their API (Entrez), however, there is already easier way by using BioPython, which I recommend.

To use it you need BioPython installed on your computer, import Entrez from BioPython

from Bio import Entrez

Simple examples are available in the documentation:

http://biopython.org/DIST/docs/api/Bio.Entrez-module.html

Entrez.email = "Your.Name.Here@example.org"
pmid = "19304878"
handle = Entrez.elink(dbfrom="pubmed", id=pmid, linkname="pubmed_pubmed")
record = Entrez.read(handle)
handle.close()

More specific instructions are in Tutorial (concerning both PubMed and MedLine):

http://biopython.org/DIST/docs/tutorial/Tutorial.html#htoc129

Entrez.email = "example@example.com"
handle = Entrez.esearch(db="pubmed", term="orchid", retmax=463)
record = Entrez.read(handle)
idlist = record["IdList"]
handle.close()

For specific information for found articles you can use Entrez.efetch using ids of articles.

handle = Entrez.efetch(db = 'pubmed', retmode = 'xml', id = idlist)
results = Entrez.read(handle)

Then, you can then handle the results as dictionaries in Python.

BioPython really made it easier to use many databases.