- General Data Exploration Tools
- Building a Python 2.7 Configuration for Scientific Programming
- Items to install at the Application level
- Library installations
- Python Components
- Utilities
- Revision History
There are a number of freely available tools for examining scientific datasets. It is far from an exhaustive list. It merely represents programs which I have actually used.
Astronomical datasets, particularly images, are usually stored in a file format called FITS (Flexible Image Transport System). FITS files can contain multiple segments, called extensions, which can contain different types of data. The PRIMARY extension is usually a two- or three-dimensional array which can contain an image, a sequence of images, or a 3-dimensional data cube.
There are two free tools useful for examining FITS data files and a third useful for exploring 3-D datasets.
SAOImage DS9
This program is best suited for viewing FITS images, from sources such as Space Telescope, SOHO, SDO, and STEREO.
FV
This program allows examination and editing of a wider range of FITS files than SAOImage DS9.
VisIt
VisIt is a tool which can read and view a wide variety of 3-dimensional datasets.
Building a Python 2.7 Configuration for Scientific Programming
Why Python?
- Free and Open source. There is no need to pay for each license or installation. This is a particularly good feature if you are running on a small cluster and can distribute the computing load across multiple processors.
- Python is an object-oriented language (Wikipedia).
- Python is an interpreted language(Wikipedia). While tending to be slower than compiled languages, in sciences, the development time is often the largest time commitment. This speeds up development. Once the program is relatively stable, many options exist for optimizing code components for faster execution.
The newest available Python as of this writing is version 3.2. However, many of the libraries I am interested in for this project are still evolving in their support for the newest version. Python 2.7 lets us have support for these libraries while providing a migration path to version 3 and above using the from __future__ import mechanism.
Why 64-bit?
In science, we often need to explore very large datasets. 64-bit processors have a much larger memory addressing space. This space is needed for data structures requiring more than 2GB of memory. You'd be surprised how quickly a 3-dimensional double-precision floating-point array can get that large.
Why build from source?
While many would argue this is a more difficult route, unless you can obtain a prepackaged installation kit that contains everything you want, configured exactly the way you want, you may encounter problems of compatibility between different installations.
It also has the advantage that you learn more about your programming environment, when can make it easier to solve other problems you may have to deal with in scientific software development.
General Instructions for SciPy on Mac
As with many scientific packages being built from source, the Apple Development Package XCode and all its compilers are required. X11 is also needed for a number of applications.
Items to install at the Application Level
Install Cmake 2.8.3 from disk image
- this installs command line tools in /usr/local/bin
- Facilitates installation of VTK
MacTeX image (dated 2010 September)
(this also installed ImageMagick 6.2.6, freetype2, ghostscript 8.51, and others)
May not be necessary but is sometimes helpful
- might want to install imageMagick as 64-bit
Python 2.7
Built-in python 2.5 & 2.6 are in /System/Library/Frameworks/ and linked from /usr/bin/
Mac Installer disk image(2.7) for OSX 10.5 and Later (x86-64)
Install with Shell profile updater to make it the default python
Adds /Library/Frameworks/Python.framework/Versions/2.7/bin to PATH
Library Installations
GNU Fortran
gfortran 4.2.3
Run the installer. The components are installed in /usr/local/bin
Make sure this is in your PATH
========================================================
zlib
Data compression library
- needed for libpng
% cd zlib-1.2.5
% export LDFLAGS="-arch x86_64"
% export FFLAGS="-arch x86_64"
% ./configure --prefix=/usr/local --64
% make
% make check
% sudo make install
========================================================
LibPNG
http://www.libpng.org/pub/png/libpng.html
Image format library
- needed for PIL, wxPython, matplotlib
% export LDFLAGS="-arch x86_64"
% export FFLAGS="-arch x86_64"
% cd libpng 1.4.4
% ./configure
% make
% make check
% sudo make install
* Careful with PATH, this can conflict with Apple's libPng.dylib, causing other applications to fail.
========================================================
libjpeg
- needed for PIL
% export LDFLAGS="-arch x86_64"
% export FFLAGS="-arch x86_64"
% cd jpeg-8c
% cp /usr/share/libtool/config/config.sub .
% cp /usr/share/libtool/config/config.guess .
% ./configure --enable-shared --enable-static
% make
% sudo mkdir -p /usr/local/man/man1
% sudo make install
=======================
libtiff
- needed for PIL
% export LDFLAGS="-arch x86_64"
% export FFLAGS="-arch x86_64"
% cd tiff-3.9.4/
% export FFLAGS="-arch x86_64"
% cd tiff-3.9.4/
% ./configure --mandir=/usr/local/share/man --with-apple-opengl-framework CFLAGS="-O2 -D_FILE_OFFSET_BITS=64" CXXFLAGS="-O2 -D_FILE_OFFSET_BITS=64"
% make
% sudo make install
================================Freetype 2.4.4
Open Source font management
% ./configure
% make
% sudo make install
================================
FFTW
Fast-Fourier Transform library
% ./configure --enable-threads \
CC="gcc -arch i386 -arch x86_64" CXX="g++ -arch i386 -arch x86_64" \
CPP="gcc -E" CXXCPP="g++ -E" F77=gfortran
% make
% sudo make install
% sudo ln -s /usr/local/fftw-3.2.2 /usr/local/fftw
Other Notes:
- Installing FFTW on Mac OS X Snow Leopard
- I'm not sure if this is still needed by numpy any more
Visualization Toolkit (VTK)
Requires X11 for visual output
VTK 5.6.1
Configure with CMake on command line
% ccmake .
BUILD_SHARED_LIBS ON
VTKMY_WRAP_PYTHON ON
VTKMY_WRAP_TCL ON
VTK_WRAP_PYTHON ON
VTK_WRAP_TCL ON
VTK_USE_CARBON OFF
VTK_USE_COCOA ON
VTK_USE_X OFF
VTK_WRAP_PYTHON ON
VTK_WRAP_TCL ON
CMAKE_INSTALL_PREFIX /usr/local
CMAKE_EXE_LINKER_FLAGS -arch x86_64
CMAKE_MODULE_LINKER_FLAGS -arch x86_64
CMAKE_SHARED_LINKER_FLAGS -arch x86_64
PYTHON_LIBRARY /Library/Frameworks/Python.framework/Versions/2.7/lib/libpython2.7.dylib
VTK_LARGE_DATA_ROOT /Users/Shared/VTKData
VTK_USE_64BIT_IDS ON
VTK_DATA_ROOT /Users/Shared/VTKData
VTK_USE_RENDERING ON
# Let it build its own library for PNG, TIFF, ZLIB, JPEG
# Make sure it finds X11 libraries (but you don't have to set it for output to X11. That is apparently handled elsewhere.
# let ccmake generate configure script
% make -j 2 # takes about an hour
% sudo make install
# install python wrapper
% cd Wrapping/Python
% python setup.py build
% sudo python setup.py install
% export DYLD_LIBRARY_PATH=/usr/local/lib/vtk-5.6/
% export LD_LIBRARY_PATH=/usr/local/lib
% export PYTHONPATH=/usr/local/lib:$PYTHONPATH
% export VTK_DATA_ROOT=/Users/Shared/VTKData
Python Components
setuptools v0.6c11
% cd setuptools-0.6c11
% python setup.py build
% sudo python setup.py install
================================wxPython
GUI for other graphics-related modules such as matplotlib
wxPython2.8-osx-unicode-py2.7
========================================================
nose 0.11.3
Testing module for numpy
% python setup.py build
% sudo python setup.py install
% python -c 'import numpy; numpy.test()'
Ran 2966 tests in 10.728s
OK (KNOWNFAIL=4, SKIP=1)
================================
Python Imaging Library, PIL 1.1.7
Modify setup.py
JPEG_ROOT = "/usr/local/lib","/usr/local/include"
ZLIB_ROOT = "/usr/lib","/usr/include"
TIFF_ROOT = "/usr/local/lib","/usr/local/include"
FREETYPE_ROOT = "/usr/local/lib","/usr/local/include"
% python setup.py build_ext -i
( gave warnings on freetype about architecture)
% python selftest.py (57 tests passed)
% sudo python setup.py install
================================
numpy
% gnutar xzf numpy-1.5.1.tar.gz
% cd numpy-1.5.1
% python setup.py build --fcompiler=gnu95
% sudo python setup.py install
% python -c 'import numpy; numpy.test()'
Ran 3006 tests in 9.662s
OK (KNOWNFAIL=4, SKIP=1)
- It is not clear if numpy still needs the LAPACK, ATLAS, FFTW libraries. Does it link to Apple's transparently or build local versions?
scipy
% cd scipy-0.9.0
% python setup.py build
% sudo python setup.py install
>>> scipy.test()
Ran 4724 tests in 71.324s
OK (KNOWNFAIL=18, SKIP=36)
matplotlib v1.01
% export CFLAGS="-O -g -arch x86_64"
% export LDFLAGS="-arch x86_64"
% gnutar xzf matplotlib-1.0.0.tar.tar.gz
% cd matplotlib-1.0.0.tar
changes in setupext.py
'_darwin' : ['/usr/local'],
in the routine,
def add_base_flags(module):
Modify add the designated lines
module.include_dirs.extend(incdirs)
module.include_dirs.append('.')
+ module.include_dirs.append('/usr/local/include')
+ module.include_dirs.append('/usr/local/include/freetype2')
module.library_dirs.extend(libdirs)
% python setup.py build
% sudo python setup.py install
Compile Matplotlib for Python on Snow Leopard - Building Matplotlib for MacOS X
- Caveat: If you have used matplotlib with an earlier version of python, you might receive errors such as "Cannot_Open_Resource" when trying to access fonts. This can be due to the paths to your older installation cached in your local '.matplotlib/' configuration directory (in your home directory). Deleting this directory so matplotlib will rebuild it from scratch seems to fix the problem.
pyfits
Module for reading and writing FITS files
% gnutar xzf pyfits-2.4.0.tar.gz
% cd pyfits-2.4.0
% python setup.py build
% sudo python setup.py install
================================
Sphinx v1.0.4
Module for documentation generation
% cd Sphinx-1.0.4
% python setup.py build
% sudo python setup.py install
After installing::
% cd doc
% sphinx-build . _build/html
POVray
Simple Open Source rendering tool.
I wanted to build this module under 64-bit, however, the configuration scripts seem to have a problem with this option. This may be a compatibility problem with the libPng installed by Apple or version 1.4.4 installed in this process. Therefore, I chose the option of forcing everything to build under 32-bit.
% cd povray-3.6.1/
% cd libraries/png
% ./configure
% make
% cd ../..
% ./configure COMPILED_BY="your name % make
% make check
% sudo make install
Some POVray user group posts provided some of the information I needed to get this to build.
Utilities
% file [filename]
Tells you what type of library or file you have: 32-bit, 64-bit, PowerPC, Intel, etc.
Acknowledgements
Hopefully I've credited all the sources I collected for assistance in making this installation work (at least for me). If you discover any errors or other issues, please e-mail me at cygnusx1@mac.com.
Revision History
2011-04-17: Initial posting
No comments:
Post a Comment