SymPy 1.1 has been released

Have you ever used sympy? AS the authors say: “SymPy is a Python library for symbolic mathematics. It aims to become a full-featured computer algebra system (CAS) while keeping the code as simple as possible in order to be comprehensible and easily extensible. SymPy is written entirely in Python.”

You can install new release by simply:

pip install -U sympy

It is promised that it will be be soon available also via conda.

What is new (among others)?

1. Many improvements to code generation, including addition of
tensorflow (to lambdify), C++, llvm JIT, and Rust language support, as
well as the beginning of AST classes.

2. Several bug fixes for floating point numbers using higher than the
default precision.

3. Improvements to the group theory module.

4. Implementation of Singularity Functions to solve Beam Bending
problems.

5. Improvements to the mechanics module.

As the main author (Aaron Meurer ) say: “A total of 184 people contributed to this release. Of these, 143 people contributed to SymPy for the first time for this release. ” Maybe you will be next?

Multiple other projects also use SymPy; just to name some, there is Cadabra for (quantum) field theory system, LaTeX Expression project for typesetting of algebraic expressions and yt for analyzing and visualizing volumetric data.

 

Official page of sympy you can find here: http://www.sympy.org/en/index.html

It is also freely available at github: https://github.com/sympy/sympy.github.com

Exact information about the release/authors/deprecations/etc. can be found here: https://github.com/sympy/sympy/wiki/Release-Notes-for-1.1

 

Advertisements

Why use Jupyter Notebook for data analyses?

I think that everyone interested in data science and data analysis somewhere, somehow during their education or internet searches comes across Jupyter Notebook.  Jupyter notebook is an aplication that enables you to create (and share) document that contains code (in various programming languages), explanaitions (text) and visualizations. Jupyter Notebook is super useful when you want to show your workflow and prepare how-to for future analyses for yourself or your team.

I use Jupyter Notebook with Python3 but you can use it with various programming languages if you prefer to. Python has very broad offer of libraries for statistical analysis, data visualizations and machine learning.

With Jupyter Notebook you can show every step of data transformation showing, e.g. pandas’ DataFrames in really nice shape:

Screen Shot 2017-07-01 at 07.53.29

Moreover you can include plots with the code you used to create it so you can easily reproduce it for other data:

Screen Shot 2017-07-01 at 07.51.01

Just to mention, super useful thing in Jupyter Notebook is:

%matplotlib inline

that makes your plots appear as you execute a cell without calling

plt.show()

I hope you see what are indisputable perks of using Jupyter Notebook, I encourage you strongly to try it out.

If you’re into Jupyter Notebook, this year there is a conference in August in New York, called Jupytercon (https://conferences.oreilly.com/jupyter/jup-ny).

There are a lot of interesting projects around Jupyter Notebook, like JupyterHub (https://jupyterhub.readthedocs.io/en/latest/) that allows Jupyter server to be used by multiple users or nteract (https://nteract.io/) that transforms Jupyter Notebooks to desktop application so it’s even easier to use.

Working with GA analytics from Python API

To analyse data from your website: users, sessions, page views, many would choose Google Analytics – GA (https://analytics.google.com/). To obtain data you can go to the GA website, choose metrics and time frame you need and download data to work with in csv or xslx format. However, there is also an API to directly import data from GA to data-analysing script (e.g. within Jupyter notebook), so why not explore it instead?

How to connect to GA API and use data from it? Great step-by-step instructions are available on GA API websites (https://developers.google.com/sheets/api/quickstart/python) and I will just combine all of needed info in one place.

First of all:

  1. Go to this page: https://console.developers.google.com/flows/enableapi?apiid=sheets.googleapis.com and create a project and (automatically) turn on the API. Click Continue, then click Go to credentials.
  2. On the Add credentials to your projectpage, click the Cancel
  3. At the top of the page, select the OAuth consent screen Select an Email address, enter a Product name if not already set, and click the Save button.
  4. Select the Credentials tab, click the Create credentials button and select OAuth client ID.
  5. Select the application type Other, enter the name and click the Create.
  6. Dismiss the resulting dialog.
  7. Click the file_download(Download JSON) button to the right of the client ID.
  8. Move this file to your working directory and rename it to client_secrets.json
  9. on your computer install google-api-python-client
    pip install --upgrade google-api-python-client

    or

  10. sudo easy_install --upgrade google-api-python-client

and you’re good to go.

Now how to connect to GA and send your first request:

(the example is based on https://github.com/EchoFUN/GAreader/blob/master/hello_analytics_api_v3.py)

from googleapiclient.errors import HttpError
from googleapiclient import sample_tools
from oauth2client.client import AccessTokenRefreshError

def ga_request(input_dict):
    service, flags = sample_tools.init(
    [], 'analytics', 'v3', __doc__, __file__,
    scope='https://www.googleapis.com/auth/analytics.readonly')

    try:
        first_profile_id = get_first_profile_id(service)
        if not first_profile_id:
            print('Could not find a valid profile for this user.')
        else:
            results = get_top_keywords(service, first_profile_id, input_dict)
            return results

    except TypeError as error:
        print(('There was an error in constructing your query : %s' % error))

    except HttpError as error:
        print(('Arg, there was an API error : %s : %s' %
              (error.resp.status, error._get_reason())))

    except AccessTokenRefreshError:
        print('The credentials have been revoked or expired, please re-run '
          'the application to re-authorize')

def get_first_profile_id(service):
    accounts = service.management().accounts().list().execute()
    if accounts.get('items'):
        firstAccountId = accounts.get('items')[0].get('id')
        webproperties = service.management().webproperties().list(
            accountId=firstAccountId).execute()

        if webproperties.get('items'):
            firstWebpropertyId = webproperties.get('items')[0].get('id')
            profiles = service.management().profiles().list(
                accountId=firstAccountId,
                webPropertyId=firstWebpropertyId).execute()

            if profiles.get('items'):
                return profiles.get('items')[0].get('id')

    return None

def get_top_keywords(service, profile_id, input_dict):
    if input_dict['filters'] == '':
        return service.data().ga().get(
            ids=input_dict['ids'],
            start_date=input_dict['start_date'],
            end_date=input_dict['end_date'],
            metrics=input_dict['metrics'],
            dimensions=input_dict['dimensions']).execute() 
    return service.data().ga().get(
        ids=input_dict['ids'],
        start_date=input_dict['start_date'],
        end_date=input_dict['end_date'], 
        metrics=input_dict['metrics'],
        filters = input_dict['filters'],
        dimensions=input_dict['dimensions']).execute()

Save this file as ga_api_example.py

You need to remember that from API, metrics and other feature names may have other exact names, e.g. custom dimensions are just called dimension[no] etc.

And now, after loading your file

from ga_api_example import ga_request

You can prepare request

request = {
"ids" : "ga:<your_id>",
"start_date" : "2017-06-25",
"end_date" : "2017-06-25",
"metrics" : "ga:pageviews",
 "filters" : "ga:dimension1=~yes",
"dimensions" : ""
}

=~ means use regex as at GA website reports

and execute it:

data = ga_request(request)

Now you have data in Python script where you work with, e.g. with pandas.

Seaborn library for pretty plots

Seaborn is visualization library based on matplotlib (and complementary to matplotlib, you should really understand matplotlib first). It basically makes your work easier and prettier. The library is not really complicated and broad but it makes some thing for you, things that you would have to do in matplotlib on your own.

Seaborn works very well with libraries used in Python for data analysis (pandas, numpy, scipy, statsmodels) and may be used easily in Jupyter notebook for plots imaging. The most frequently mentioned advantage of seaborn are built-in themes. Well… it helps people who don’t know how to combine different colors to make plots aesthetic. Second thing is that functions do really nice plots that try to show something useful even when called with a minimal set of arguments. However, as for matplotlib you have almost endless number of possibilities to adjust your plots.

Additional information about seaborn can be find here: https://seaborn.pydata.org/

Installation is pretty easy:

pip install seaborn

or

conda install seaborn

Additionall info about installation (including development version of seaborn) can be found here: http://seaborn.pydata.org/installing.html

They did really nice job also when comes to documentation and tutorials (e.g. https://seaborn.pydata.org/tutorial.html)

My favorite thing about seaborn? I would say seaborn.distplot function. I usually do two visualizations to look at data before working with it – scatter plot and distribution. Scatter plot is probably easy to obtain in any visualization lib you work with, as is with matplotlib. However to see distribution along with KDE plot, I recommend seaborn function displot.

Here are some examples: https://seaborn.pydata.org/generated/seaborn.distplot.html?highlight=distplot#seaborn.distplot

Basically, all you need to do to have your data in array, e.g. as pandas DataFrame column

import seaborn as sns
ax = sns.distplot(df['column name'])

To try with example data, you can try plotting normally distributed data:

import seaborn as sns, numpy as np
x = np.random.randn(100)
ax = sns.distplot(x)

 

Testing modules within cherrypy

I was wondering how unittests may be implemented within cherrypy to test not cherrypy itself but the modules included. I wanted tests to be run every time when new changes were introduced, that means when the server reloads.

There are some information about unittesting in cherrypy (e.g. https://stackoverflow.com/questions/14260101/unittesting-cherrypy-webapp), however, none directly addressed my issue.

I decided to prepare single file named unittests.py (remember not to name your file as unittest.py as there will be problem with library import) with unittests and TextTestRunner setup. TextTestRunner does not interrupt the reload of the server as standard unittest.main() does.

import unittest
from module import function1, function2

class MyTestClass(unittest.TestCase):

    def test_search(self):
        self.assertEqual(function1(input1), output1)
        self.assertEqual(function2(input2), output2)

def start():
    suite = unittest.TestSuite()
    suite.addTest(MyTestClass("test_search"))
    runner = unittest.TextTestRunner()
    runner.run(suite)

and in the main cherrypy file, besides import, I added single line (unittests.start()):

import unittests

if __name__ == "__main__":
    cherrypy.config.update("config.conf")
    unittests.start()
    cherrypy.quickstart(Page(), config="config.conf")

Every time the server reloaded the tests were run and I could check whether everything is correct.

Python in Big Data #1 – Hadoop & Snakebite

One of the often mentioned term during searching for Big Data information is Hadoop. What is Hadoop exactly? And what are first steps to handle Hadoop with Python?

Hadoop is a system file (HDFS) that enable scalable data handling. Basically, Hadoop was developed to store large amount of data while providing reliability and scalability. It is based on data block system. HDFS is based on two processes: NameNode that gathers metadata and DataNodes that store blocks of data. Blocks of data are replicated on different machines to provide data stability when one machine crash.

There are increasing number of libraries for Python that enables handling Hadoop. Just to name some: Snakebite, mrjob, PySpark.

Snakebite enables basic file operations on Hadoop and accessing Hadoop from Python applications. It can be install easily:

pip install snakebite

While installed, within Python application you can connect with HDFS NameNode with:

from snakebite.client import Client
client = Client('localhost', 8000)

Then you can implement various functions similar to functions in shell: ls, mkdir, delete to handle files and directories within Hadoop. Another group of functions (e.g. copyToLocal) enable retrieving data from HDFS.

from snakebite.client import Client
client = Client('localhost', 8000)
for file in client.copyToLocal(['/hdfs_1/file1.txt'], '/tmp'):
    print file

Snakebite provide also CLI client.

Some additional information can be found in a free book:

Hadoop with Python by Zachary Radtka and Donald Miner

http://www.oreilly.com/programming/free/hadoop-with-python.csp

How Python connects you with biological databases? #2 – PubMed

Pubmed is the biggest database of biological scientific papers. There are other databases that gather information about all scientific papers (e.g. google scholar, scopus), however in biological sciences, still most commonly used is Pubmed from NCBI.

There is quite easy way to access Pubmed through their API (Entrez), however, there is already easier way by using BioPython, which I recommend.

To use it you need BioPython installed on your computer, import Entrez from BioPython

from Bio import Entrez

Simple examples are available in the documentation:

http://biopython.org/DIST/docs/api/Bio.Entrez-module.html

Entrez.email = "Your.Name.Here@example.org"
pmid = "19304878"
handle = Entrez.elink(dbfrom="pubmed", id=pmid, linkname="pubmed_pubmed")
record = Entrez.read(handle)
handle.close()

More specific instructions are in Tutorial (concerning both PubMed and MedLine):

http://biopython.org/DIST/docs/tutorial/Tutorial.html#htoc129

Entrez.email = "example@example.com"
handle = Entrez.esearch(db="pubmed", term="orchid", retmax=463)
record = Entrez.read(handle)
idlist = record["IdList"]
handle.close()

For specific information for found articles you can use Entrez.efetch using ids of articles.

handle = Entrez.efetch(db = 'pubmed', retmode = 'xml', id = idlist)
results = Entrez.read(handle)

Then, you can then handle the results as dictionaries in Python.

BioPython really made it easier to use many databases.