01 Scientific modules and IPython

This is part of Python for Geosciences notes.

================

Core scientific packages

When people say that they do their scientific computations in Python it's only half true. Python is a construction set, similar to MITgcm or other models. Without packages it's only a core, that although very powerful, does not seems to be able to do much by itself.

There is a set of packages, that almost every scientist would need:

We are going to talk about all exept Sympy

IPython and pylab

In order to be productive you need comfortable environment, and this is what IPython provide. It was started as enhanced python interactive shell, but with time become architecture for interactive computing.

Starting IPython with --pylab option loads some necessary modules (NumPy for data array support and Matplotlib for plotting support), that make it similar to Matlab console.

In order to use latest version of IPython on zmaw computers one has to load appropriate module:

module load python/2.7-ve2

and then

ipython --pylab

On Ubuntu or other Debian-based distributions something like this will set you up for the rest of the course:

sudo apt-get install ipython-notebook python-matplotlib python-scipy python-pandas python-sympy python-mpltoolkits.basemap netcdf-bin cdo

IPython notebook

Since the 0.12 release, IPython provides a new rich text web interface - IPython notebook. Here you can combine:

Code execution

In [1]:
print('I love Python')

I love Python

Text (Markdown)

IPython website.

List:

Code:

print('hello world')

$\LaTeX$ equations

$$\int_0^\infty e^{-x^2} dx=\frac{\sqrt{\pi}}{2}$$ $$ F(x,y)=0 ~~\mbox{and}~~ \left| \begin{array}{ccc} F''_{xx} & F''_{xy} & F'_x \\ F''_{yx} & F''_{yy} & F'_y \\ F'_x & F'_y & 0 \end{array}\right| = 0 $$

Plots

In [2]:
x = [1,2,3,4,5]
plot(x)
Out[2]:
[]

Rich media

In [36]:
from IPython.display import YouTubeVideo
YouTubeVideo('F4rFuIb1Ie4')
Out[36]:

Run notebook

In order to start IPython notebook you have to type:

ipython notebook

but in order to work in the notebook efficiently one has to start it with pylab and inline mode, that allow inline graphics

ipython notebook --pylab inline

on ZMAW computers, where default browser is Konqueror, you might need to use:

ipython notebook --pylab inline --browser="firefox"

You can download and run this lectures:

Web version can be accesed from the github repository.

You can download them as .zip file:

wget https://github.com/koldunovn/python_for_geosciences/archive/master.zip

Unzip:

unzip master.zip

And run:

cd python_for_geosciences-master/
ipython notebook --pylab inline

Main IPyhton features

Getting help

You can use question mark in order to get help. To execute cell you have to press Shift+Enter

In [1]:
?

Question mark after a function will open pager with documentation. Double question mark will show you source code of the function.

In [6]:
plot??

Press TAB after opening bracket in order to get help for the function (list of arguments, doc string).

In [ ]:
sum(

Accessing the underlying operating system

You can access system functions by typing exclamation mark.

In [7]:
!pwd

/home/magik/NOTEBOOKS/PYTHON

If you already have some netCDF file in the directory and ncdump is installed, you can for example look at its header.

In [8]:
!ncdump -h test_netcdf.nc

netcdf test_netcdf {
dimensions:
	TIME = 1464 ;
	LATITUDE = 73 ;
	LONGITUDE = 144 ;
variables:
	float TIME(TIME) ;
		TIME:units = "hours since 1-1-1 00:00:0.0" ;
	float New_air(TIME, LATITUDE, LONGITUDE) ;
		New_air:missing_value = -9999.f ;
	float LONGITUDE(LONGITUDE) ;
	float LATITUDE(LATITUDE) ;
}

Example of cdo use:

In [9]:
!cdo nyear test_netcdf.nc

1
cdo nyear: Processed 1 variable over 1464 timesteps. ( 0.01s )

Get information from OS output to the python variable

In [10]:
nmon = !cdo nmon test_netcdf.nc
nmon
Out[10]:
['cdo nmon: Processed 1 variable over 1464 timesteps. ( 0.01s )', '12']

Return information from Pyhton variable to the SHELL

In [11]:
!echo {nmon[1]}

12

Magic functions

The magic function system provides a series of functions which allow you to control the behavior of IPython itself, plus a lot of system-type features.

Let's create some set of numbers using range command:

In [6]:
range(10)
Out[6]:
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

And find out how long does it take to run it with %timeit magic function:

In [14]:
%timeit range(10)

1000000 loops, best of 3: 679 ns per loop

Print all interactive variables (similar to Matlab function):

In [7]:
%whos

Interactive namespace is empty.

Cell-oriented magic

Receive as argument both the current line where they are declared and the whole body of the cell.

In [15]:
%%timeit
range(10)
range(100)

100000 loops, best of 3: 3.34 µs per loop

Thre are several cell-oriented magic functions that allow you to run code in other languages:

In [16]:
%%bash

echo "My shell is:" $SHELL
echo "My memory status is:"
free

My shell is: /bin/bash
My memory status is:
             total       used       free     shared    buffers     cached
Mem:       3925620    1857860    2067760          0      82812     757212
-/+ buffers/cache:    1017836    2907784
Swap:      3906556          0    3906556
In [14]:
%%perl

$variable = 1;
print "The variable has the value of $variable\n";

The variable has the value of 1

You can write content of the cell to a file with %%writefile (or %%file for ipython < 1.0):

In [16]:
%%writefile hello.py
#if you use ipython < 1.0, use %%file comand
#%%file 
a = 'hello world!'
print(a)

Writing hello.py

And then run it:

In [17]:
%run hello.py

hello world!

The %run magic will run your python script and load all variables into your interactive namespace for further use.

In [18]:
%whos

Variable   Type    Data/Info
----------------------------
a          str     hello world!

In order to get information about all magic functions type:

In [19]:
%magic

Links:

The cell magics in IPython

Comments !

links

social