View source for Montage

= Montage =

* Doc v1.3

* Note: fits files can be open with the program skycat

== Compile Montage binaries ==

* Note: remember to install 'libc6-dev-amd64'

* Note: for Montage_v3.0, remember to add '-lpthread' and '-ldl' options in Montage_v3.0/grid/Pegasus/Makefile.LINUX

<pre>
wget http://montage.ipac.caltech.edu/download/Montage_v3.3.tar.gz
tar -xzf Montage_v3.3.tar.gz
cd Montage_v3.3
make
cd ..
</pre>

== Run Montage over m101 galaxy data ==

* From http://montage.ipac.caltech.edu/docs/m101tutorial.html

=== Prepare the test ===

<pre>
mkdir test_m101_from_irsa
cd test_m101_from_irsa
PATH=../Montage_v3.3/bin:$PATH
</pre>

=== Prepare input data in 'fits' directory ===

<pre>
mkdir input_dir
cd input_dir
</pre>

=== Download astronomic data ===

* Go to ' http://irsa.ipac.caltech.edu/applications/2MASS/IM/batch.html '

use the first form

<pre>
>   Retrieve 'ATLAS' images 'overlapping' a Region
>   All Sky Release Survey Images
>   Region Center =                 M 101
>   Region Diameter =               0.5 // original tutorial is a rectangle of 0.2 degree area, sqr(0.2) = 0.447
>   Decompress Quicklook images  =  false // less data to download
>   Band =                          J // in the montage benckmark file the J band is the only defined
</pre>

* Save and extract the archives

<pre>
tar -xf m101_0.5degree_j.tar
gzip -d *.fits.gz
</pre>

=== Create image definition file ".hdr" ===

* Go to 'http://irsa.ipac.caltech.edu/applications/HdrTemplate/'

use

<pre>
>   Object or location:     M 101
>   System:                 Equatorial
>   Equinox:                2000.
>   Width:                  0.5
>   Height:                 0.5
>   Resolution:             1.0
>   Rotation:               0.0
</pre>

* Write the result into 'template.hdr' file

* Go back to the base directory

<pre>
cd ..
</pre>

=== Reproject images ===

<pre>
mkdir metadata_dir
</pre>

* Create a .tbl file describing your input data

<pre>
mImgtbl input_dir metadata_dir/input_fits.tbl
</pre>

* Execute reprojecting

<pre>
mkdir reprojected_dir
mProjExec -p input_dir metadata_dir/input_fits.tbl input_dir/template.hdr reprojected_dir metadata_dir/reprojected_stats.tbl
</pre>

* Recreate a .tbl file describing your generated reprojected data

<pre>
mImgtbl reprojected_dir metadata_dir/reprojected.tbl
</pre>

=== Generating first version of 'uncorrected' mosaic ===

<pre>
mkdir output_dir
mAdd -p reprojected_dir metadata_dir/reprojected.tbl input_dir/template.hdr output_dir/mosaic_uncorrected.fits
mJPEG -gray output_dir/mosaic_uncorrected.fits 20% 99.98% loglog -out output_dir/mosaic_uncorrected.jpg
</pre>

=== Smooth out the background levels ===

* Calculate the best way to smooth out the overlap regions using the difference images

<pre>
mOverlaps metadata_dir/reprojected.tbl metadata_dir/diffs.tbl
mkdir diff_dir
mDiffExec -p reprojected_dir metadata_dir/diffs.tbl input_dir/template.hdr diff_dir
mFitExec metadata_dir/diffs.tbl metadata_dir/plane_fitting.tbl diff_dir
</pre>

* Apply the background removal to the original reprojected images

<pre>
mBgModel metadata_dir/reprojected.tbl metadata_dir/plane_fitting.tbl metadata_dir/background_corrections.tbl
mkdir background_corrections_dir
mBgExec -p reprojected_dir metadata_dir/reprojected.tbl metadata_dir/background_corrections.tbl background_corrections_dir
</pre>

=== Correct mosaic ===

<pre>
mAdd -p background_corrections_dir metadata_dir/reprojected.tbl input_dir/template.hdr output_dir/mosaic_corrected.fits
mJPEG -gray output_dir/mosaic_corrected.fits 0s max gaussian-log -out output_dir/mosaic_corrected.jpg
cd ..
</pre>

== Run Montage Benchmark (serial run) ==

=== Get benchmark data ===

<pre>
wget http://sukhna.isi.edu/wf_benchmark/pegasus_wf_benchmark.tgz
</pre>

=== Get DAG python api ===

<pre>
mkdir Pegasus
wget https://raw.githubusercontent.com/pegasus-isi/pegasus/master/lib/pegasus/python/Pegasus/DAX2.py -P Pegasus
touch Pegasus/__init__.py
</pre>

=== Create the file 'pegasus_dependencies_generator.py' with this content ===

<pre>

"""
Extract the list of commands serially equivalents to the input Pegasus workflow, in the correct order.

v 1.3
"""

import argparse
import Pegasus.DAX2


class OrderedSet():

    def __init(self):
        self.list = list()
        self.set = set()

    def __init__(self, *data):
        self.list = list()
        self.set = set()

        for val in data:
            self.push(data)

    def __getitem__(self, index):
        return self.list[index]

    def __len__(self):
        return len(self.set)

    def push(self, value):
        if value not in self.set:
            self.list.append(value)
            self.set.add(value)

    def pop(self, value):
        if value in self.set:
            self.list.remove(value)
            self.set.remove(value)

    def has(self, value):
        return value in self.set

    def __contains__(self, value):
        return value in self.set


class PegasusDependenciesGenerator:

    def __init__(self, input, output):
        self.identified_jobs = OrderedSet()
        self.adag = Pegasus.DAX2.parse(input)
        self.output = open(output, "w")

        try:
            # save all jobs in order of dependency
            for job in self.adag.jobs:

                # if job is in the list, all dependencies are too
                if job not in self.identified_jobs:
                    # retrieve dependency jobs
                    dep_jobs = self.getDependencyOrderedJobs(job)

                    # save dep_job to be writed if not was already done
                    for dep_job in dep_jobs:
                        if dep_job not in self.identified_jobs:
                            self.identified_jobs.push(dep_job)

                    # save the job itself
                    if job not in self.identified_jobs:
                        self.identified_jobs.push(job)

            # write saved jobs
            for job in self.identified_jobs:
                self.writeJob(job)

        finally:
            self.output.close()

    def getDependencyOrderedJobs(self, job_to_check):

        dep_jobs = OrderedSet()

        if job_to_check in self.adag.lookup:
            dependencies = self.adag.lookup[job_to_check]

            for dep_job in dependencies.parents:
                sub_dep_jobs = self.getDependencyOrderedJobs(dep_job)

                for sub_dep_job in sub_dep_jobs:
                    if sub_dep_job not in dep_jobs:
                        dep_jobs.push(sub_dep_job)

        return dep_jobs

    def writeJob(self, job):

        out = OrderedSet()
        out.push(job.name)

        for arg in job.arguments:
            if isinstance(arg, str) or isinstance(arg, unicode):
                out.push(arg)
            elif isinstance(arg, Pegasus.DAX2.Filename):
                out.push(arg.filename)
            else:
                raise Exception("getting type " + str(type(arg)) + " : " + arg)

        self.output.write(" ".join(out) + "\n")


if __name__ == "__main__":

    parser = argparse.ArgumentParser()
    parser.add_argument(
        "input", type=str, help="the input dax XML file")
    parser.add_argument(
        "output", type=str, help="the output dependencies file")
    args = parser.parse_args()

    pg = PegasusDependenciesGenerator(args.input, args.output)
    # job = pg.adag.lookup.keys()[10]
    # print (pg.getDependencyOrderedJobs(job))


</pre>

=== Create the script 'pegasus_run.sh' to support the run ===

<pre>
#!/bin/sh

INPUT=inputdata;
BIN=../Montage_v3.3/bin;
COMMANDS=dependencies.txt;

test ! -d ${INPUT} && echo >&2 "Error: no '${INPUT}' folder, exit" && exit 1;
test ! -d ${BIN} && echo >&2 "Error: no '${BIN}' folder, exit" && exit 1;
test ! -f ${COMMANDS} && echo >&2 "Error: no '${COMMANDS}' file, exit" && exit 1;

PATH=$PATH:../${BIN};

cd ${INPUT};
cat ../${COMMANDS} | while read CMD; do
    /bin/echo -e "${CMD}\t...";
    ${CMD};
    test "$?" -ne 0 && echo "Error, exit" && exit 1;
    /bin/echo -e "Done.";
done;

# End
</pre>

=== Generate the list of commands and run them sequencially ===

<pre>
mkdir test_montage_benchmark
cd test_montage_benchmark
tar -xzf ../pegasus_wf_benchmark.tgz
python ../pegasus_dependencies_generator.py pegasus_wf_benchmark/montage/montage-1.0-dax/montage-1.0.dax dependencies.txt
tar -xzf pegasus_wf_benchmark/montage/input_data.tgz
time sh ../pegasus_run.sh
cd ..
</pre>

== Generate a Pegasus workflow of m101 galaxy for 0.5x0.5 degree of area of the sky ==

* From https://confluence.pegasus.isi.edu/display/pegasus/Montage

<pre>
PATH=`pwd`/Montage_v3.3/bin:$PATH
mkdir test_m101_benchmark_0.5x0.5
cd test_m101_benchmark_0.5x0.5
mkdir dag_dir
mDAG 2mass j M101 0.5 0.5 0.000278 dag_dir "gsiftp://sukhna.isi.edu`pwd`" "gsiftp://sukhna.isi.edu`pwd`/input"
mkdir inputdata
cd inputdata
mArchiveExec ../dag_dir/images.tbl
gzip -d *.fits.gz
cd ..
test -f inputdata.tgz || tar -czf inputdata.tgz inputdata  # save a backup of downloaded data
python ../pegasus_dependencies_generator.py dag_dir/dag.xml dependencies.txt
cp dag_dir/big_region.hdr inputdata/big_region_20150928_002641_303.hdr
cp dag_dir/statfile.tbl inputdata/statfile_20150928_002641_303.tbl
cp dag_dir/pimages.tbl inputdata/pimages_20150928_002641_303.tbl
cp dag_dir/cimages.tbl inputdata/cimages_20150928_002641_303.tbl
cp dag_dir/region.hdr inputdata/region_20150928_002641_303.hdr
(nohup time sh ../pegasus_run.sh) &> serial-run.log &
</pre>

== Generate a Pegasus workflow of m101 galaxy for 6x6 degree of area of the sky ==

* From https://confluence.pegasus.isi.edu/display/pegasus/Montage

<pre>
PATH=`pwd`/Montage_v3.3/bin:$PATH
mkdir test_m101_benchmark_6x6
cd test_m101_benchmark_6x6
mkdir dag_dir
mDAG 2mass j M101 6 6 0.000278 dag_dir "gsiftp://sukhna.isi.edu`pwd`" "gsiftp://sukhna.isi.edu`pwd`/input"
mkdir inputdata
cd inputdata
mArchiveExec -d 1 ../dag_dir/images.tbl
cd ..
test -f inputdata.tar || tar -cf inputdata.tar inputdata  # save a backup of downloaded data
cd inputdata
gzip -d *.fits.gz
cd ..
python ../pegasus_dependencies_generator.py dag_dir/dag.xml dependencies.txt
cp dag_dir/big_region.hdr inputdata/big_region_20150928_011927_8886.hdr
cp dag_dir/statfile.tbl inputdata/statfile_20150928_011927_8886.tbl
cp dag_dir/pimages.tbl inputdata/pimages_20150928_011927_8886.tbl
cp dag_dir/cimages_0_0.tbl inputdata/cimages_0_0_20150928_011927_8886.tbl
cp dag_dir/cimages_0_1.tbl inputdata/cimages_0_1_20150928_011927_8886.tbl
cp dag_dir/cimages_0_2.tbl inputdata/cimages_0_2_20150928_011927_8886.tbl
cp dag_dir/cimages_1_0.tbl inputdata/cimages_1_0_20150928_011927_8886.tbl
cp dag_dir/cimages_1_1.tbl inputdata/cimages_1_1_20150928_011927_8886.tbl
cp dag_dir/cimages_1_2.tbl inputdata/cimages_1_2_20150928_011927_8886.tbl
cp dag_dir/cimages_2_0.tbl inputdata/cimages_2_0_20150928_011927_8886.tbl
cp dag_dir/cimages_2_1.tbl inputdata/cimages_2_1_20150928_011927_8886.tbl
cp dag_dir/cimages_2_2.tbl inputdata/cimages_2_2_20150928_011927_8886.tbl
cp dag_dir/region_0_0.hdr inputdata/region_0_0_20150928_011927_8886.hdr
cp dag_dir/region_0_1.hdr inputdata/region_0_1_20150928_011927_8886.hdr
cp dag_dir/region_0_2.hdr inputdata/region_0_2_20150928_011927_8886.hdr
cp dag_dir/region_1_0.hdr inputdata/region_1_0_20150928_011927_8886.hdr
cp dag_dir/region_1_1.hdr inputdata/region_1_1_20150928_011927_8886.hdr
cp dag_dir/region_1_2.hdr inputdata/region_1_2_20150928_011927_8886.hdr
cp dag_dir/region_2_0.hdr inputdata/region_2_0_20150928_011927_8886.hdr
cp dag_dir/region_2_1.hdr inputdata/region_2_1_20150928_011927_8886.hdr
cp dag_dir/region_2_2.hdr inputdata/region_2_2_20150928_011927_8886.hdr
cp dag_dir/slist.tbl inputdata/slist_20150928_011927_8886.tbl
cp dag_dir/shrunken.hdr inputdata/shrunken_20150928_011927_8886.hdr
(nohup time sh ../pegasus_run.sh) &> serial-run.log &
</pre>