Firas Sadiyah

Python Exception Handling

Sat, 24 Feb 2024 12:00:00 +0000

In genomic data analysis, we often use a pipeline function to process data stored in a dataframe by calling several mini-functions. Each mini-function may modify the dataframe by adding a new column with new values and then filter out the rows that do not meet certain criteria. However, this may result in an empty dataframe if none of the rows satisfy the filters and can lead to errors or unexpected results when the pipeline function tries to perform more operations on the empty dataframe. To avoid this situation, we can use two strategies. First, we can check if the DataFrame is non-empty before applying any logic in each mini-function. Second, we can make the pipeline function fail graciously if it receives an empty DataFrame from any of the mini-functions by using a custom exception and a try-except block. Let’s take a look.

Directing Python Output in Terminal

Sun, 18 Feb 2024 12:00:00 +0000

When writing Python scripts, it’s common to want to display information messages in the user terminal and possibly save them to a file like output.log. Therefore, it’s essential to understand the levels of logging in Python, the types of outputs in Unix-like systems, and how to direct the appropriate type of logging to the right output.

Understanding Output Streams in Unix

In Unix systems, there are two primary streams for output: standard output (stdout) and standard error (stderr). These streams serve different purposes:

Running PyEnsembl on CentOS HPC

Sun, 11 Feb 2024 12:00:00 +0000

Today, I am installing a Python package on one of the high-performance computing (HPC) systems that I work with. This CentOS 7 setup resembles a fortress, with its stringent security protocols, understandable due to hosting clinical data, but making it quite a challenge to install essential Python packages from public repositories.

The package I’m attempting to install is PyEnsembl, a handy tool for accessing Ensembl’s genetic information.

PyEnsembl Setup

I started by creating a new conda environment:

Debugging Python codebases using PyCharm and VSCode

Sun, 04 Feb 2024 12:00:00 +0000

Debugging, an essential process for identifying and rectifying errors in a computer program, is particularly crucial for computational biologists dealing with complex codebases that often involve intricate mathematical models, data analysis, and simulations. Merely reading the code may not suffice to grasp the logic and functionality of the project. To gain a deeper understanding, you may need to run the code, examine the variables, and observe the outputs. Towards this end, the integrated debuggers in PyCharm and VSCode prove invaluable.

Managing Python virtual environments

Sun, 28 Jan 2024 12:00:00 +0000

When it comes to Python development on macOS, I rely on a combination of two tools that have served me exceptionally well over the past few years: Pyenv and Poetry. Pyenv provides an elegant solution for managing different Python versions on my system, while Poetry simplifies dependency management and the creation of virtual environments for my projects. In this article, I will guide you through the process of setting up Pyenv to install a specific Python version and then using Poetry to create a virtual environment for your project.

Running NetMHCPan on Apple Silicon

Tue, 23 Jan 2024 12:00:00 +0000

NetMHCPan, a widely used tool for predicting peptide binding to major histocompatibility complex (MHC) molecules, is essential for understanding immune responses. However, the tool’s binaries are currently available only for the x86_64 architecture, whether on Darwin (macOS) or Linux. As I intended to conduct test runs on my Apple Silicon device (arm64), I encountered the following error:

1netMHCpan: no binaries found for Darwin_arm64 /net/sund-nas.win.dtu.dk/storage/services/www/packages/netMHCpan/4.1/netMHCpan-4.1/Darwin_arm64/bin/netMHCpan

To address this, one option is to run NetMHCPan using Rosetta, a dynamic binary translator that translates executable code on-the-fly. In simple terms, Rosetta intercepts instructions intended for one architecture (Intel, in this case) and converts them into instructions compatible with another architecture (Apple Silicon).

Setting up a reproducible R environment on macOS

Mon, 10 Oct 2022 12:00:00 +0000

Using renv is an excellent choice for maintaining a clean and reproducible R environment on macOS. Here, I will share my experiences and provide a guide on setting up R on macOS. The post is divided into the following sections:

Installing system dependencies required for R libraries using Homebrew.
Installing R libraries using renv.
Saving and restoring R environments using renv.
Installing R libraries hosted on private repositories.
Building R libraries from source using Makevars.

Let’s get started!

Running NextFlow using AWS Batch

Sat, 01 May 2021 00:00:00 +0000

Processing Genomics workflows require a variety of tools with distinct computing resources requirements. Some of these requirements are merely beyond the configuration of powerful workstations. High-performance clusters (HPC) offer the possibility to scale up computing resources to meet the increasing demands of these processes. However, maintaining the infrastructure of an on-premises HPC may not be a viable solution for many parities. A more cost-effective method is to provide computer resources dynamically by leveraging cloud computing. This strategy, combined with a powerful workflow manager such as NextFlow, allow researchers to execute their workflows locally and/or in the cloud with minimal friction.

Using reticulate in RStudio with pyenv

Wed, 28 Apr 2021 00:00:00 +0000

When developing in Python, it is generally a good practice not to rely on the Python version that ships with the operating system (OS). This is to ensure that the system version of Python remains relatively ‘clean’ for the OS processes. In addition, by installing a custom version(s) of Python, we open many possibilities. For one, it gives us control over which specific version(s) to use in our projects, and two, by using a virtual environment manager, we ensure that each project has access to its own tailored list of packages. One way of achieving this is by using pyenv.

Using data.table with OpenMP support

Mon, 26 Apr 2021 00:00:00 +0000

If you are facing difficulties with large data sets in R, using data.table could provide a performance boost. However, when loading data.table, especially on macOS, you might encounter a warning indicating the absence of OpenMP support, causing data.table to operate in a single-threaded mode. This limitation prevents you from fully utilizing the potential benefits of using data.table and taking advantage of the underlying hardware.

 1library(data.table)
 2data.table 1.14.0 using 1 threads (see ?getDTthreads).  Latest news: r-datatable.com
 3**********
 4This installation of data.table has not detected OpenMP support.
 5It should still work but in single-threaded mode.
 6If this is a Mac, please ensure you are using R>=3.4.0 and have followed our Mac instructions here:
 7https://github.com/Rdatatable/data.table/wiki/Installation.
 8This warning message should not occur on Windows or Linux. If it does, please file a GitHub issue.
 9**********
10
11getDTthreads()
12[1] 1

What is the issue here?

OpenMP is an implementation of multithreading, and the Clang compiler that comes with Xcode on macOS lacks support for OpenMP. Apple has chosen not to include the libomp.dylib run-time library in their compiler. You can verify this by executing the following command.

Using RegEx to capture file name in Groovy/NextFlow

Sat, 06 Oct 2018 12:00:00 +0000

My sequencing files are named according to the folowing pattern lane5651_AAGAGGCA_00h_Cell_WT3_L008_R1.fastq.gz. I would like to capture the Sample ID as 00h_Cell_WT3 in order to name all downstream files accordingly. To this end, I wrote the following snippet:

 1#!/usr/bin/env nextflow
 2
 3// fastq files are stored in reads as paired ends R1 and R4
 4params.reads = 'reads/lane*_*_*_*_R{1,4}.fastq.gz'
 5
 6Channel
 7     .fromFilePairs(params.reads, flat: true)
 8     .map { prefix, file1, file2 -> tuple(getSampleID(prefix), file1, file2) }
 9     .set { samples_ch }
10
11def getSampleID( file ) {
12    // using RegEx to extract the SampleID
13    // in paried ends, fromFilePairs (with flat: true) returns a triplate
14    //     where the first item is the filename without `R{1,4}.fastq.gz`
15    //     thus the RegEx needs to be adjusted as follow
16    regexpPE = /([a-z]{4}[0-9]{4})_([A-Z]{8})_(.+)_(L[0-9]{3})/
17    (file =~ regexpPE)[0][3]
18}
19
20process printNames {
21    input:
22    set sampleId, forward_reads, reverse_reads from samples_ch
23
24    output:
25    stdout result
26
27    """
28    echo $sampleId 'and' $forward_reads 'and' $reverse_reads
29    """
30}
31
32result.subscribe { println it }

For single ends experiments, the following snippet can be used:

An overview of Illumina multiplexing

Sun, 10 Jun 2018 12:00:00 +0000

Multiplexing

Index 1 (i7) is always read.
Index 2 (i5) is read only in dual index setting.

	Single indexing	Dual indexing
Single end (SE)	R1, R2	R1, R2, R3
Paired end (PE)	R1, R2, R4	R1, R2, R3, R4

Multiplexing - dual index reads - paired end

Preparing genome reference in FASTA format

Sun, 10 Dec 2017 12:00:00 +0000

To prepare genome reference in FASTA format for mouse assembly NCBI37/mm9, we have two options:

From UCSC

Using the mm9 assembly from UCSC golden Path.
Do not use the masked file chromFaMasked.tar.gz!

 1# download `chromFa.tar.gz ` from UCSC golden path
 2wget http://hgdownload.cse.ucsc.edu/goldenPath/mm9/bigZips/chromFa.tar.gz
 3#or
 4sync -avzP rsync://hgdownload.cse.ucsc.edu/goldenPath/mm9/bigZips/chromFa.tar.gz .
 5
 6# uncompress the downloaded file
 7tar -xvzf chromFa.tar.gz
 8
 9# remove `*random.fa` chromosomes
10rm -rf *_random.fa
11
12# concatenate all FASTA files into a single file
13cat *.fa > mm9.fa
14
15# index the concatenated .fa file using `samtools`
16module load samtools # required at HPC
17samtools faidx mm9.fa

From Ensembl

Using Mus Musculus release-67 (Ensembl)

 1# download the following folder from Ensembl
 2lftp ftp://ftp.ensembl.org/pub/release-67/fasta/mus_musculus/dna/
 3mget *
 4
 5# uncompress the downloaded *.fa.gz files
 6gzip -d *.fa.gz
 7
 8# delete the masked version of the genome sequence which contains '_rm' in the name
 9rm -rf *_rm*
10
11# concatenate all FASTA files into a single file
12cat *.fa > mm9Ensembl.fa
13
14# index the concatenated .fa file using `samtools`
15module load samtools # required at HPC
16samtools faidx mm9Ensembl.fa

One of the main differences between the two sources, which is very important for downstream applications, is chromosome annotation.
Here is a comparison between the header lines, also known as the identifier or description lines, used in the FASTA files from both sources.
While UCSC uses the chr prefix in front of the chromosome number, Ensembl merely uses the chromosome number.

 1# mm9 from UCSC
 2~/f/G/N/mm9_fasta ❯❯❯ cat mm9.fa | grep '>'
 3>chr10
 4>chr11
 5>chr12
 6>chr13
 7>chr14
 8>chr15
 9>chr16
10>chr17
11>chr18
12>chr19
13>chr1
14>chr2
15>chr3
16>chr4
17>chr5
18>chr6
19>chr7
20>chr8
21>chr9
22>chrM
23>chrX
24>chrY
25
26# ENS67 from Ensembl
27~/f/G/N/ENS67_fasta ❯❯❯ cat ENS67.fa | grep '>'
28>10 dna:chromosome chromosome:NCBIM37:10:1:129993255:1
29>11 dna:chromosome chromosome:NCBIM37:11:1:121843856:1
30>12 dna:chromosome chromosome:NCBIM37:12:1:121257530:1
31>13 dna:chromosome chromosome:NCBIM37:13:1:120284312:1
32>14 dna:chromosome chromosome:NCBIM37:14:1:125194864:1
33>15 dna:chromosome chromosome:NCBIM37:15:1:103494974:1
34>16 dna:chromosome chromosome:NCBIM37:16:1:98319150:1
35>17 dna:chromosome chromosome:NCBIM37:17:1:95272651:1
36>18 dna:chromosome chromosome:NCBIM37:18:1:90772031:1
37>19 dna:chromosome chromosome:NCBIM37:19:1:61342430:1
38>1 dna:chromosome chromosome:NCBIM37:1:1:197195432:1
39>2 dna:chromosome chromosome:NCBIM37:2:1:181748087:1
40>3 dna:chromosome chromosome:NCBIM37:3:1:159599783:1
41>4 dna:chromosome chromosome:NCBIM37:4:1:155630120:1
42>5 dna:chromosome chromosome:NCBIM37:5:1:152537259:1
43>6 dna:chromosome chromosome:NCBIM37:6:1:149517037:1
44>7 dna:chromosome chromosome:NCBIM37:7:1:152524553:1
45>8 dna:chromosome chromosome:NCBIM37:8:1:131738871:1
46>9 dna:chromosome chromosome:NCBIM37:9:1:124076172:1
47>MT dna:chromosome chromosome:NCBIM37:MT:1:16299:1
48>X dna:chromosome chromosome:NCBIM37:X:1:166650296:1
49>Y dna:chromosome chromosome:NCBIM37:Y:1:15902555:1

macOS setup for data science

Sun, 08 Oct 2017 12:00:00 +0000

Here, I summarise how I setup my macOS for the purpose of Data Science. Ultimately, I should invest some time into automating the process.

Install Command Line Developer Tools

1xcode-select --install

Install HomeBrew

1/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Turn off analytic (optional)

1brew analytics off

Install iTerm

1brew install iterm

Configure iTerm

To download different colour schemes, visit iTerm Themes

Install Go2Sehll

Download and install from Go2Shell, then configure it to work with iTerm:

Ubuntu setup for data science

Sat, 16 Sep 2017 12:00:00 +0000

This posts shows how I setup a new Ubuntu for Data Science.

Update Ubuntu

1sudo apt update
2sudo apt upgrade

Configure apt

It’s always a good idea to check the baseline setup. So, let’s check the current apt keys.

1sudo apt-key list

Then, let’s check the sources.

1cat /etc/apt/sources.list
2cd /etc/apt/sources.list.d/

And finally, let’s check the users and group.

 1cut -d: -f1 /etc/passwd # will list all local users
 2cat /etc/passwd # will list all local users with groups and other properties
 3getent passwd # same as above
 4
 5cut -d: -f1 /etc/group # will list all local groups
 6cat /etc/group # will list all local groups
 7id -Gn
 8groups
 9
10getent group groupname # will list all members of groupaname
11id username # will list all the groups a particular username belongs to

Configure Unity

To get minimise window on clicking its icon:

Mounting remote drives locally using sshfs

Tue, 29 Aug 2017 12:00:00 +0000

To mount a network drive as a local one, you can use sshfs.

Generate SSH key

1$ mkdir ~/.ssh #if it does not already exist
2$ chmod 700 ~/.ssh
3$ cd ~/.ssh
4$ ssh-keygen -t rsa
5$ enter a keyname
6$ enter a passphrase
7$ ssh-copy-id -i [path to rsa file] USER@SERVER

Where -i indicates where the rsa file is located. The .pub key is the one needed to be copied.

[Optional] Start a SSH agent, and add the key to the SSH agent:

Managing Python virtual environments

Tue, 01 Aug 2017 12:00:00 +0000

Installation

 1# macOS
 2brew install python3 python-pip
 3
 4# Ubuntu
 5sudo apt install python3 python-pip
 6
 7# Update pip to the latest version
 8pip install --upgrade pip setuptools wheel
 9pip3 install --upgrade pip setuptools wheel
10
11# Install virtualenvwrapper
12pip3 install virtualenv virtualenvwrapper

Create projects directory

1mkdir ~/Projects

Configure .profile (Bash) or .zprofile (ZSH):

1# needed for virtualenvwrapper
2export WORKON_HOME=$HOME/.virtualenvs
3export PROJECT_HOME=$HOME/Projects
4export VIRTUALENVWRAPPER_PYTHON=/bi/home/USER/.linuxbrew/bin/python3
5export VIRTUALENVWRAPPER_VIRTUALENV=/bi/home/USER/.linuxbrew/bin/virtualenv
6export PIP_REQUIRE_VIRTUALENV=true
7source /bi/home/USER/.linuxbrew/bin/virtualenvwrapper.sh

Reload profile

1source ~/.profile
2source ~/.zprofile

Make a new project

The command will create a python virtual environment by running mkvirtualenv.
Then it will creates a project directory and will turn to that directory.

1# using the default python version
2mkproject projectName
3
4# using specific python version
5mkproject -p `which python` projectName

To delete a project

1rmvirtualenv projectName

Bonus

If you are using ZSH shell with prezto (recommended), you can change prezto prompt to indicate the virtual environment currently in use. You need to edit prompt_sorin_setup as follow¹ :

DaPars for alternative polyadenylation analysis

Sun, 19 Mar 2017 12:00:00 +0000

A colleague of mine asked me for help in using DaPars for analysing alternative polyadenylation in their RNA-seq dataset. So, I thought to write a short post here to describe how I use it.

From Xia et al. 2014

Here we develop a novel bioinformatics algorithm (DaPars) for the de novo identification of dynamic APAs from standard RNA-seq.

Installation

Download the source files of DaPars from GitHub and extract the files:

1$ tar zxvf DaPars-VERSION.tar.gz

Input files

You can find more details on the documentation page, but in essence, DaPars requires the following files: