1.1.1   2006-10-29 module pdb

• a quick tutorial: http://www.ferg.org/papers/debugging_in_python.html
• It's better to put the testing code in a file, rather than pasting it into the python console. Pasting disables the l(list) command. You'll have trouble to locate the line position of the codes.
• In order to skip big chunk of for-loop in a function, insert pdb.set_trace() right before what you want and in the python console, import the function and type pdb.run("function()"). After the pdb console shows up, type s to step into the function. The function will stop right before the trace.

1.1.2   2008-02-20 try except

sys.exc_info() returns a tuple of except type, except description and a traceback object.

import traceback
try:
  print var
except:
  sys.stderr.write('Except type: %s\n'%repr(sys.exc_info()[0]))
  traceback.print_exc()
  print sys.exc_info()
  #raise sys.exc_info()[0]
  sys.exit(2)

1.1.3   2008-02-28 garbage collection

To free up more memory, try following:

import gc
gc.collect()    #run a full collection
gc.set_threshold(50,2,2)   #Set the garbage collection thresholds (the collection frequency). Setting threshold0 to zero disables collection.
del gc.garbage[:]   #delete uncollectable objects

gc.set_threshold(50,2,2) incurs the danger of slowing down python due to frequent collection.

1.1.4   2008-03-31 useful functions in debug mode

locals() and globals() show the variables in the current environment.

1.2.1   2008-01-14 gnome canvas documentation

library reference manual It's for C language. But seems there's no documentation for python-gnomecanvas and the package is just a .so file.

nice example

Book Chapters (outdated, but useful to understand the idea behind gnome canvas)

http://developer.gnome.org/doc/books/WGA/index.html

http://developer.gnome.org/doc/GGAD/ggad.html

1.2.2   2008-01-21 python-gtk/python-gnome documentation

python-gtk(pygtk): the tutorial that comes with it. Its tutorial also appears on its website along with its reference.

python-gnome: http://library.gnome.org/devel/references. The C library references. It also includes references of other libraries related to Gnome. Can't find python-gnome's own documentation. It appears as .so/dynamic-libraries and comes with only examples.

1.2.3   2008-01-21 existing glade projects in gtk/gnome

• annot/bin/guianalyzer.glade
• annot/bin/graph/genepairgui.glade
• utility/grid_job_mgr/grid_job_mgr.glade
• variation/src/Genotyping/QCVisualizeII.glade
• variation/src/QCVisualize.py (not by glade, pygtk + matplotlib)
• proj200303/script/pygnome (not by glade, gtk/gnome ...)

1.2.4   2008-01-21 AppBar

3 choices:

gnome.ui.AppBar

the glade 'Gnome App Bar' widget

the AppBar embedded in the glade 'Gnome App' widget

Only the embedded AppBar works as the C library references show. gnome.ui.AppBar gives out GtkWarning: gtk_label_set_text: assertion 'GTK_IS_LABEL (label)' failed. The glade 'Gnome App Bar' widget doesn't have correct APIs.

2008-02-12. import gnome.ui has to be executed before using its push() method, self.app1_appbar1.push('Status Message.').

1.2.5   2008-01-23 widget's properties and user custom data

dir(widget.props) shows all properties' names. and widget.props.property_name gives out the value of that property.

widget.get_properties('property_name') or widget.get_property('property_name') is another way to get property value. The former gives a tuple (possible multiple property values for one property)

widget.set_properties('property_name', 'property_value') or widget.set_property('property_name', 'property_value') or widget.set(property_name='property_value') sets that property.

user custom data

w.__dict__ is a dictionary to be used by user. w.__getattribute__('dict-key') fetches the value in that dictionary.

w.set_data(data_name, data_value) and w.get_data(data_name) is a another way.

1.2.6   2008-02-01 redirect the stdout/stderr output to a TextView widget

One way is to write a dummy file class which has all necessary APIs for it to be a file object but writes the output to a text buffer. source is http://www.daa.com.au/pipermail/pygtk/attachments/20031004/dbb34c38/Py_Shell.py. Implemented in pymodule.gnome

2nd way is to embed a terminal into your GUI. http://ubuntuforums.org/archive/index.php/t-366720.html. didn't implement this one.

1.2.7   2008-11-07 type coercion in gtk.ListStore

Upon initialization, types of all columns will be specified in gtk.ListStore. If a specific column's type is not str, feeding it with a None would raise an exception. Has to feed it with values of appropriate type. Like if it's int, feed it a 0.

Although it raises an exception, it can still handle the data inserted into the ListStore and display it. Just catch the exception to avoid program to break down.

1.3.1   2008-11-07 avoid deadlock

1. default for subprocess.Popen() is no_wait. If you intend to wait, as cp_p.wait(), it might deadlock the whole program.
2. repetitive calling of subprocess.Popen() + cp_p.stderr.read() could deadlock the program easily, esp when stdout/stderr is redirected to a gtk TextView widget. For some mysterious reasons, cp_p.stderr.read() ahead of cp_p.stdout.read() increases the chance of deadlock. stdout_content, stderr_content = cp_p.communicate() is a much stable call than stderr.read() + stdout.read().

1.5.1   @property

Put right above a function in a class. It declares this method to be accessed as an attribute.:

class a(object):
@property
def my_func_var(self):
  return 100

a_ins = a()
print a_ins.my_func_var

2008-05-09 @property not available in python2.3. use my_func_var=property(my_func_var) to replace it.

1.5.2   chr(...)

chr(i) -> character

Return a string of one character with ordinal i; 0 <= i < 256.

1.6.1   a python script

If a python script is in dos format and you edit it in linux and add #!/usr/bin/python on the top to make it executable, when you run it directly, it'll report error : No such file or directory. Because it keeps the dos format although you modify it in linux. Run dos2unix solves it.

1.6.2   data file

If a data file is in dos format, python program will read the line separator as an extra character although it could recognize the end of line. Run dos2unix.

1.7.1   2008-05-29 python data structures that pass by reference

• list []
• dictionary {}

1.7.2   2008-05-29 class initialization for these data structures

Example 1:

>>> class a:
...         my_ls = []
...         def __init__(self, my_dict={}, my_other_dict=None):
...                 pass
... 
>>> a_ins = a()
>>> b_ins = a()
>>> a_ins.my_ls.append('a')
>>> print b_ins.my_ls
['a']

Example 2:

>>> class a:
...         my_ls = None
...         def __init__(self, my_dict=None, my_other_dict=None):
...                 pass
... 
>>> a_ins = a()
>>> b_ins = a()
>>> a_ins.my_ls = []
>>> a_ins.my_ls.append('a')
>>> print b_ins.my_ls
None
>>> a_ins.my_dict = {}

Basically initialization of those pass-by-reference data structure will cause different instances of the same class share the same data structure unless you re-evaluate the variable after instantiation.

1.9.1   2008-11-28 numpy.nan

WATCH: numpy.nan!=numpy.nan. Must use numpy.isnan() to judge whether a value is nan or not.

1.9.2   2009-10-3 upon conversion into numpy.array, empty list ([]) is casted into numpy.float64

clearly visible from:

>>> numpy.array([])
array([], dtype=float64)

It would spell trouble if you are trying to concatenate two lists together, one is empty and the other is integer list. It would cast the type of the combined array to float, which would cause trouble if it's used as an index list later on:

>>> numpy.hstack(([], [1,2,3]))
array([ 1.,  2.,  3.])

1.10.1   2009-11-21 percentile, quartile, median

get any percentile:

from scipy import stats
minimum = numpy.min(intensity_ls)
first_decile = stats.scoreatpercentile(intensity_ls, 10)
lower_quartile = stats.scoreatpercentile(intensity_ls, 25)
median = stats.scoreatpercentile(intensity_ls, 50)
upper_quartile = stats.scoreatpercentile(intensity_ls, 75)
last_decile = stats.scoreatpercentile(intensity_ls, 90)

2.3.1   Documentation

http://www.sqlalchemy.org/docs/. Watch the version difference between the documentation and the SQLAlchemy library you are using.

http://www.rmunn.com/sqlalchemy-tutorial/tutorial.html

http://spyced.blogspot.com/2007/01/why-sqlalchemy-impresses-me.html

2.3.2   example codes

check repositary code test/python/test_sqlalchemy.py for examples etc.

If you wanna create a table-corresponding object and put it into database later, you either have an __init__() to have attributes initialized or assign attributes after the object is instantiated.:

>>> class a(object):
...  def __init__(self, name=None):
...         self.name = name
... 
>>> a_o = a(name='1st object')
>>> print a_o.name
1st object
>>> 
>>> class b(object):
...         pass
... 
>>> (initialize table mapper ...)
>>> b_o = b(name='1st ojbect')  #won't work. b_o.name is None
>>> print b_o.name
None
>>> b_o = b()
>>> b_o.name = '1st object'
>>> print b_o.name
1st object
>>> 

2.3.3   208-05-23 access table field in sql.select()

Two forms exist, either thru the original Table() object, or the python class object. However, the python class object doesn't work always. In test_sqlalchemy.py, both works. In variation repository, the python class object doesn't. Might be related where the Mapper() is called.

example:

#by Table() object
a = Table(...)
sql.select([a.c.id,a.c.name])
...
class A(object):
  ...
Mapper(A, a, ...)
#by python class object
sql.select([A.c.id, A.c.name])

Wait to find out what causes this ...

2.3.4   2008-05-24 sqlalchemy's wrapper: collective.lead

in collective/lead/database.py, in _initialize_engine()

Bug 1: when new db engine is created, old session (if there's one) does not close or bind to the new engine. This causes the session still uses the old db setting (_url). add these code in it:

if getattr(self._threadlocal, 'session', None) is not None:   #2008-05-24 added by yh. new engine binding
    self._threadlocal.session.close()
self._threadlocal.session = Session(bind=engine)

Bug 2: self.tables[name] = table.tometadata(self._metadata). self._metadata hasn't been updated to the new metadata. so self.tables[name] = table.tometadata(metadata).

2.3.5   2008-05-07 order_by

Putting it as a function of the whole select, select([addresses]).order_by(addresses.c.id), according to http://www.sqlalchemy.org/docs/04/sqlexpression.html, gives error:

File "/usr/lib/python2.5/site-packages/sqlalchemy/engine/base.py", line 519, in execute
  raise exceptions.InvalidRequestError("Unexecuteable object type: " + str(type(object)))
sqlalchemy.exceptions.InvalidRequestError: Unexecuteable object type: <type 'NoneType'>

order_by is a keyword argument of sqlalchemy.sql.select() by looking at sqlalchemy/sql.py:

select([addresses], order_by=[addresses.c.id])

2.3.6   2008-05-30 memory management

One problem with the sqlalchemy's session thing is that it'll hoard up lots of objects in memory (and memory usage is huge compared to plain sql) before a final commit.

session.clear() or session.close() after transaction commit doesn't help to release memory. Even deleting connection. maybe need to delete objects.

2008-07-10 this is no longer a problem in version 0.4. after session.flush() and session.clear(), memory is cleared up.

2.3.7   2008-07-10 set table schema for postgres db

specifying "schema=" in Table() solves it. some people suggests con=engine.connect(); con.execute("set search_path to ..."); metadata.bind=con. it doesn't work.:

table1 = Table(table_name, metadata, autoload=True, schema=getattr(self, 'schema', None))

In elixir, either specify using_table_options in Entity-derived class:

class Gene(Entity):
  ...
  using_table_options(schema='public')

OR use using_table_options_handler() to set it later:

from elixir.options import using_table_options_handler
for entity in entities:
  using_table_options_handler(entity, schema=self.schema)

2.3.8   2008-07-10 relationship between tables

In establishing the map (SQlAlchemy) or defining the class (elixir), make sure the python keyword and the table column would be different (default). If you customize the column name same as the keyword, the python keyword takes precedence and overwrites the column name. In your python application, that same name has to refer to a python object (which represents a row in another table).:

class Gene2go(Entity):
      tax_id = Field(Integer)
      gene = ManyToOne('Gene', colname='gene_id', ondelete='CASCADE', onupdate='CASCADE')
      ...

2.3.9   2008-07-17 how to autocommit in elixir 0.5

omitting the whole session.begin()-session.commit() pair means autocommit because the default is autoflush=True, autocommit=False in create_session() of sqlalchemy/orm/__init__.py.

• Rumour is in 0.6 it's changed. check http://elixir.ematia.de/trac/wiki/Migrate05to06:

  session in elixir = sqlalchemy.orm.scoped_session(session_maker()).
  

• assuming autoflush=True, autocommit=False (with SA 0.5 -- equivalent to transactional=True with SA 0.4), autoexpire=True.

2.3.10   2008-08-07 how to do raw sql query in elixir

The metadata data structure imported from elixir has an attribute bind, which is equivalent to sqlalchemy's connection.:

from elixir import metadata
...
metadata.bind = ... #connect to a database
rows = metadata.bind.execute("select * from table")
row = rows.fetchone()
for row in rows:
  print row.id

2008-08-17 according to http://elixir.ematia.de/trac/wiki/FAQ#HowdoIexecuteSQLdirectly, this should work but did not.:

from elixir import *
for result in session.execute('select * from people'):
    print result['name']

['_ResultProxy__ambiguous_processor', '_ResultProxy__echo', '_ResultProxy__keys', '_ResultProxy__props', 
'__class__', '__delattr__', '__dict__', '__doc__', '__getattribute__', '__hash__', '__init__', '__iter__',
'__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__str__', '__weakref__',
'_create_key_cache', '_fetchall_impl', '_fetchmany_impl', '_fetchone_impl', '_get_col', '_has_key', 
'_init_metadata', '_key_cache', '_process_row', '_rowcount', 'close', 'closed', 'connection', 'context', 
'cursor', 'dialect', 'fetchall', 'fetchmany', 'fetchone', 'keys', 'last_inserted_ids', 'last_inserted_params', 
'last_updated_params', 'lastrow_has_defaults', 'lastrowid', 'out_parameters', 'postfetch_cols', 'rowcount', 
'scalar', 'supports_sane_multi_rowcount', 'supports_sane_rowcount']

2.3.11   2008-08-12 how to access table name in elixir

After defining a table class inheriting from Entity, say MyTable, you can access its table name via MyTable.table.name.

2.3.12   2008-08-26 Working with Multiple Databases

Elixir has a link showing how to do that, http://elixir.ematia.de/trac/wiki/Recipes/MultipleDatabases.

Program test/python/test_elixir.py showcases how to use the 2nd approach (multiple files, defining __metadata__ and __session__).

The trick is that setup_all() needs to be invoked only once and after all __metadata__ are set up.

2.3.13   2009-5-26 search by regular expression

in sqlalchemy, use column.op('regexp')(...):

table1.select(table1.c.column1.op('regexp')('pattern'))

a bit tricky to do it in elixir:

rows = Gene_symbol2id.query.filter_by(tax_id=tax_id).filter(Gene_symbol2id.table.c.gene_symbol.op('regexp')('^%s'%namelike))

2.7.1   2008-12-31 installation

the version coming with ubuntu is 0.9.6, missing some features mentioned in pylons book. so install 0.9.7 manually.

1. remove some python packages to avoid conflict:

   sudo apt-get remove python-pylons paste-common python-beaker python-formencode python-mako
   python-nose python-paste python-pastedeploy 
   python-pastescript python-pastewebkit python-routes python-simplejson python-webhelpers
   

2. install 0.9.7 throu easy_install, -a is to ensure --always-copy (-a) Copy all needed packages to install dir in case some packages are already registered in /usr/lib/python2.5/site-packages/easy-install.pth, which might be broken due to apt-get remove:

   crocea@banyan:~$ sudo easy_install -a -f http://pylonshq.com/download/0.9.7 -U Pylons
   

   all packages installed: Pylons, Tempita 0.3, WebTest 1.1, WebError 0.10, WebOb 0.9.4, Mako 0.2.4, nose 0.10.4, simplejson 2.0.6, FormEncode 1.2.1, PasteScript 1.7.3, PasteDeploy 1.3.2, Paste 1.7.2, Beaker 1.1.2, WebHelpers 0.6.4, Routes 1.10.1, Pygments 1.0. complete log is pylons_package_install.log/view.

3. 2008-12-31 Elixir 0.6 doesn't work with the web project because in defining relationships (like OneToMany, ManyToOne, etc.) (quote):

   Default "target entity resolving code" changed slightly. It now uses a global collection keyed on the entity name. This means that entities can refer to other entities in a different module simply with the target entity name instead of its full path. The full path is only required when there is an ambiguity (ie when there are two classes with the same name in two different modules).

   apt-get remove python-elixir and sudo easy_install --always-unzip elixir==0.5.2

   2009-11-17 Elixir 0.6 works now. The full path means "variation.src.Stock_250kDB.README" or "variation.src.StockDB.README" rather than "README". The path depends on how the module is imported. A universal way around it is "%s.README"%__name__ in which __name__ is the module name (= __main__ if it's running as a standalone program).

2.7.2   2008-12-31 Minimal Steps to run a 0.9.6 project under 0.9.7

http://wiki.pylonshq.com/display/pylonsdocs/Upgrading

1. Add the following lines to config/environment.py:

   # Add these imports to the top
   from beaker.middleware import CacheMiddleware, SessionMiddleware
   from routes.middleware import RoutesMiddleware
   
   # Add these below the 'CUSTOM MIDDLEWARE HERE' line, or if you removed
   # that, add them immediately after the PylonsApp initialization
   app = RoutesMiddleware(app, config['routes.map'])
   app = SessionMiddleware(app, config)
   app = CacheMiddleware(app, config)
   

2. The default behavior of the 'c' object has changed in 0.9.7, to the 'strict_c' behavior which will throw an exception if an attribute is accessed of 'c' that doesn't exist. The pre-0.9.7 behavior can be restored by adding the following to config/environment.py:

   # After config.init_app:
   config['pylons.strict_c'] = False
   

3. Action arguments are no longer attached to 'c' by default as well, to restore the old implicit behavior add the following line to config/environment.py:

   # After config.init_app:
   config['pylons.c_attach_args'] = True
   

4. The Rails helpers from WebHelpers are no longer automatically imported in the webhelpers package. To use them 'lib/helpers.py' should be changed to import them:

   from webhelpers.rails import *
   

5. Your Pylons 0.9.6 project should now run without issue in Pylons 0.9.7. Note that some deprecation warnings will likely be thrown reminding you to upgrade other parts.

2.7.3   2009-1-30 install google-visualization-python

AKA Google visualization Python API.

1. download from http://code.google.com/p/google-visualization-python/
2. untar ...
3. python setup.py install

2.7.4   2009-4-1 plugins for firefox (debug purpose)

• Firebug http://getfirebug.com/. esp for javascript debugging
• LiveHTTPHeaders http://livehttpheaders.mozdev.org/

2.7.5   2009-3-4 quick paster commandlines

• create a new project named "HelloWorld":

  paster create --template=pylons HelloWorld
  paster create -t pylons HelloWorld
  

• list all templates:

  paster create --list-templates
  

• start the server:

  paster serve --reload development.ini
  

• create a controller:

  paster controller hello
  

• interactive shell:

  paster shell test.ini
  paster shell development.ini
  

2.7.6   2009-1-29 set form default value

If form helpers is used, no way to set the default of entries in the form. Only through the controller's defaults.:

defaults = {'call_method_id': id,
                "remove_old_plots":"on"}
c.call_method_ls = self.getCallMethodLs()
c.call_method_ls.insert(0, [0, u'Please Choose ...'])
html = render('/display_results_form.html')
return htmlfill.render(html, defaults)

2.7.7   2009-4-3 jsonify encoding error

@jsonify on top of controller method would automatically convert your whatever python data structure into json data structure but it assumes encoding utf-8, which would fail when dealing with swedish letters (latin1 encoding). so uncomment it and do it manually:

#@jsonify       #2009-4-3, the default encoding 'utf-8' doesn't work due to some swedish letters.
def autoComplete(self):
        """
        2009-4-3
        """
        namelike = request.params.get('namelike')
        name_ls = self.findAccessionsNameLike(namelike)
        name_ls.sort()
        if len(name_ls)>100:
                name_ls = name_ls[:100]
        return simplejson.dumps(dict(result=name_ls), encoding='latin1')

2.7.8   2009-5-27 unquote/decode URL

URL are usually encoded/quoted in the browser window and between data passing. Replace special characters in string using the %xx escape. Letters, digits, and the characters '_.-' are never quoted.

So if a parameter from a URL request contains special characters, it needs to be unquoted beforehand, which is to replace %xx escapes by their single-character equivalent:

import urllib
geneName = urllib.unquote(request.params.get('geneName'))

2.7.9   2009-7-22 route map

map between route path and controller/action could override the controller-based maps. For example:

map.connect('/', controller='hello', action='index')
....
map.connect('/{controller}')
map.connect('/{controller}/')
map.connect('/{controller}/{action}')
map.connect('/{controller}/{action}/')
map.connect('/{controller}/{action}/{id}')

h.url_for(controller='hello'), h.url_for(controller='hello', action='index') and h.url_for(controller='hello', action='index', id=None) all map to the same path, '/', rather than '/hello' or '/hello/index'. Unless you change the id to 'home', anything different from the default None. Another way is to put direct path like /hello there to utilize the controller-based maps.

Under usual circumstances, h.url_for(controller='hello') and / are same under the route map. It could become different when / is redirected/rewritten to something else by apache.

2.7.10   2009-6-23 character encoding in HTML

according to http://en.wikipedia.org/wiki/Character_encodings_in_HTML, characters below were encoded.

1. &amp; → & (ampersand, U+0026)
2. &lt; → < (less-than sign, U+003C)
3. &gt; → > (greater-than sign, U+003E)
4. &quot; → " (quotation mark, U+0022)
5. &apos; → ' (apostrophe, U+0027)

The encoding happens during the argument passing from controller to the template via c.. In the template, a simple javascript line would transform it back.:

var short_name = "${c.short_name}";
short_name = short_name.replace(/&amp;/g, '&').replace(/&lt;/g, '<').replace(/&gt;/g, '>');

However, the encoding happens again when the variable is read from the html(template) into a GWT js variable. So in GWT java code, this is necessary:

phenotypeMethodShortName = phenotypeMethodShortName.replaceAll("&lt;", "<").replaceAll("&gt;", ">").replaceAll("&amp;", "&");
// OR
phenotypeMethodShortName = URL.decodeComponent(phenotypeMethodShortName); // not verified

2.8.1   installation

easy_install doesn't work because pyjamas would be installed in /usr/share/pyjamas, which is outside easy_install's directories. seems no way around it. both -d dir and -S dir failed.

• download http://pypi.python.org/packages/source/P/Pyjamas/Pyjamas-0.5.tar.gz (probably same as from code.google). tag pyjamas_0_5 from svn repositary has installation problem. python setup.py install misses lots of ui files and others.

• create ./bin/buildout:

  sudo python bootstrap.py
  

• build pyjsbuild pyjscompile:

  ./bin/buildout
  

• install libraries (stuff installed into /usr/share/pyjamas/):

  sudo python setup.py install
  

Note:

• import logging as log was inserted in the beginning of /usr/lib/python2.5/site-packages/setuptools/command/sdist.py. somehow, log.warn() in sdist.py is visited and log module is not found. probably a bug in setuptools.

2.8.2   TroubleShooting

#from pyjamas.uiObject import EventObject
from pyjamas.ui.EventObject import EventObject

def __sendRequest(self, method, params, handler):
     id = pygwt.getNextHashId()
     msg = {"id":id, "method":method, "params":params}
     msg_data = self.parser.encode(msg)  #turn into json
     request_info = JSONRequestInfo(id, method, handler)
     #if not HTTPRequest().asyncPost(self.url, msg_data, JSONResponseTextHandler(request_info)):
     #2009-3-31 asyncPost() doesn't work. pylons never receives the post data. use asyncGet() below instead.
     if not HTTPRequest().asyncGet(self.url+'?json=%s'%msg_data, JSONResponseTextHandler(request_info)):
         return -1
     return id

def __setitem__(self, key, value):
    JS("""
    var sKey = this._keyToStr(key);
    this.d[sKey]=value;     //2009-4-1 used to be [key,value]. this change might break down others.
    """)

def __getitem__(self, key):
    JS("""
    var sKey = this._keyToStr(key);
    var value=this.d[sKey];
    return value;   //2009-4-1 used to be value[1]
    """)

2.8.3   Pyjamas-Desktop http://pyjd.org/

Desktop version of Pyjamas.