[{"content":"In genomic data analysis, we often use a pipeline function to process data stored in a dataframe by calling several mini-functions. Each mini-function may modify the dataframe by adding a new column with new values and then filter out the rows that do not meet certain criteria. However, this may result in an empty dataframe if none of the rows satisfy the filters and can lead to errors or unexpected results when the pipeline function tries to perform more operations on the empty dataframe. To avoid this situation, we can use two strategies. First, we can check if the DataFrame is non-empty before applying any logic in each mini-function. Second, we can make the pipeline function fail graciously if it receives an empty DataFrame from any of the mini-functions by using a custom exception and a try-except block. Let\u0026rsquo;s take a look.\nWhat is Exception Handling? At times, Python code encounters errors that halt program execution. These errors, known as exceptions, arise for various reasons. For instance, attempting to divide a number by zero triggers a ZeroDivisionError exception. To prevent the program from crashing, we can use exception handling techniques to deal with the errors in different ways. For example, we can show a clear message to the user, write the error details to a log file, or try a different approach to solve the problem.\nOne way to handle exceptions in Python is to use the try-except statement. This statement allows us to test a block of code for possible errors and execute another block of code when an error occurs. The general syntax of the try-except statement is:\n1try: 2 # code that might cause an error 3except: 4 # code to handle the error We can also specify the type of error we want to handle after the except keyword. This way, we can have different blocks of code for different errors. For example, we can handle the ZeroDivisionError and the ValueError separately:\n1try: 2 # code that might cause a ZeroDivisionError or a ValueError 3except ZeroDivisionError: 4 # code to handle the ZeroDivisionError 5except ValueError: 6 # code to handle the ValueError In addition, we can specify our own custom error types if we need to handle more specific errors, as below.\nMinimal Example In the following minimal example, process_vcf() is a pipeline function that calls several mini-functions, each of which may return an empty dataframe. Let\u0026rsquo;s explore how we can apply the concept of exception handling to manage the situation where a mini-function returns an empty dataframe.\n1import pandas as pd 2 3# create a dataframe with some genome variants 4vcf_df = pd.DataFrame({\u0026#34;chrom\u0026#34;: [\u0026#34;chr1\u0026#34;, \u0026#34;chr2\u0026#34;, \u0026#34;chr3\u0026#34;, \u0026#34;chr4\u0026#34;], 5 \u0026#34;start\u0026#34;: [1000, 2000, 3000, 4000], 6 \u0026#34;end\u0026#34;: [1010, 2010, 3010, 4010], 7 \u0026#34;ref\u0026#34;: [\u0026#34;A\u0026#34;, \u0026#34;C\u0026#34;, \u0026#34;G\u0026#34;, \u0026#34;T\u0026#34;], 8 \u0026#34;alt\u0026#34;: [\u0026#34;T\u0026#34;, \u0026#34;G\u0026#34;, \u0026#34;C\u0026#34;, \u0026#34;A\u0026#34;]}) 9 10def filter_by_region(df): 11 # filter by a region of interest, this operation will return an empty dataframe 12 roi_chrom = \u0026#34;chr1\u0026#34; 13 roi_start = 0 14 roi_end = 5000 15 # filter the dataframe by the chromosome and the overlapping range 16 return df[(df[\u0026#34;chrom\u0026#34;] == roi_chrom) \u0026amp; (df[\u0026#34;start\u0026#34;] \u0026lt; roi_end) \u0026amp; (df[\u0026#34;end\u0026#34;] \u0026lt; roi_start)].copy() 17 18def calculate_vaf(df): 19 # calculate the variant allele frequency 20 df.loc[:, \u0026#34;vaf\u0026#34;] = df[\u0026#34;alt\u0026#34;].apply(lambda x: x.count(\u0026#34;,\u0026#34;) + 1) / (df[\u0026#34;ref\u0026#34;].apply(len) + df[\u0026#34;alt\u0026#34;].apply(len)) 21 return df 22 23def process_vcf(df): 24 df = filter_by_region(df) 25 df = calculate_vaf(df) 26 # ... more mini functions 27 print(df) 28 29process_vcf(vcf_df) Adding Check for Empty DataFrame The first strategy is to check if the dataframe is non-empty before applying any logic in each mini-function. This can be achieved using the .empty property of the dataframe, which returns True if the dataframe has no rows or columns, and False otherwise.\n1def calculate_vaf(df): 2 # calculate the variant allele frequency 3 if not df.empty: # check if the dataframe is empty 4 df.loc[:, \u0026#34;vaf\u0026#34;] = df[\u0026#34;alt\u0026#34;].apply(lambda x: x.count(\u0026#34;,\u0026#34;) + 1) / (df[\u0026#34;ref\u0026#34;].apply(len) + df[\u0026#34;alt\u0026#34;].apply(len)) 5 return df This way, we can avoid performing any operations on an empty dataframe that may cause errors or unexpected results.\nImplementing Try-Except The second strategy is to make the pipeline function fail graciously if it receives an empty dataframe from any of the mini-functions by using 1) a custom exception and 2) a try-except block.\nA custom exception is a user-defined class that inherits from the base Exception class and allows us to create our own type of exception. For example, we can create a custom exception called EmptyDataFrameException as follows:\n1class EmptyDataFrameException(Exception): 2 pass This class will simply pass the message that we want to display when the exception is raised.\nNow, let\u0026rsquo;s integrate this custom exception into our functions. First, we\u0026rsquo;ll modify the calculate_vaf() function to raise EmptyDataFrameException when the dataframe becomes empty:\n1def calculate_vaf(df): 2 # calculate the variant allele frequency 3 if not df.empty: # check if the dataframe is empty 4 df.loc[:, \u0026#34;vaf\u0026#34;] = df[\u0026#34;alt\u0026#34;].apply(lambda x: x.count(\u0026#34;,\u0026#34;) + 1) / (df[\u0026#34;ref\u0026#34;].apply(len) + df[\u0026#34;alt\u0026#34;].apply(len)) 5 return df 6 # If dataframe is empty after operation, raise an exception 7 if df.empty: 8 msg = \u0026#34;calculating variant allele frequency resulted in an empty dataframe\u0026#34; 9 raise EmptyDataFrameException(msg) 10 11 return df If the input dataframe is empty, it will raise an EmptyDataFrameException with a clear message that indicates the source of the error.\nNext, we\u0026rsquo;ll modify the process_vcf() function to catch the EmptyDataFrameException and handle it gracefully. We can use a try-except block to wrap the code that may raise the exception. If the exception occurs, we\u0026rsquo;ll print the message, return the empty dataframe without any further processing. For example:\n1def process_vcf(df): 2 try: 3 df = filter_by_region(df) 4 df = calculate_vaf(df) 5 # ... more mini functions 6 print(df) 7 except EmptyDataFrameException as e: 8 print(f\u0026#34;Exiting the program due to {e}\u0026#34;) 9 return df This code will try to execute the pipeline function and print the final dataframe. However, if any of the mini-functions raises an EmptyDataFrameException, it will catch it and print the message that explains the source of the error. This way, we can avoid any further errors or unexpected results that may occur due to the empty dataframe.\nElse and Finally In addition to the try and except blocks, we can also use the else and finally blocks to handle different scenarios when an exception occurs. The else block is used to execute some code when no exception occurs in the try block. The syntax of the try-except-else statement is:\n1try: 2 # code that may cause an exception 3except: 4 # code to run when an exception occurs 5else: 6 # code to run when no exception occurs Let\u0026rsquo;s explore how we can incorporate them into our genomic example by introducing a new function that utilises the else block to print a message upon successful completion of an operation. Additionally, this example illustrates the use of multiple except statements, which handle different error types including ZeroDivisionError, ValueError, and KeyError.\n1def calculate_allele_frequency(df, allele, pop_size): 2 # calculate the allele frequency of a variant in the dataframe, given a population size 3 try: 4 # check if the population size is positive 5 if pop_size \u0026lt;= 0: 6 # raise a ValueError if the population size is zero or negative 7 raise ValueError(\u0026#34;The population size must be positive.\u0026#34;) 8 # get the number of chromosomes with the allele 9 allele_count = df[\u0026#39;ref\u0026#39;].str.count(allele).sum() + df[\u0026#39;alt\u0026#39;].str.count(allele).sum() 10 # calculate the allele frequency 11 allele_freq = allele_count / pop_size 12 # print the allele frequency 13 print(f\u0026#34;The allele frequency of {allele} is {allele_freq:.4f}.\u0026#34;) 14 except ZeroDivisionError: 15 # print an error message if the population size is zero 16 print(\u0026#34;The population size cannot be zero.\u0026#34;) 17 except ValueError as e: 18 # print the error message if the population size is negative 19 print(e) 20 except KeyError: 21 # print an error message if the allele is not valid 22 print(f\u0026#34;The allele {allele} is not valid. It must be one of A, C, G, or T.\u0026#34;) 23 else: 24 # print a message when no exception occurs 25 print(\u0026#34;The calculation was successful.\u0026#34;) This way, we can provide feedback to the user when the operation is interrupted due to any of the errors defined above, as well as when it is completed without any errors.\nFinally, we can use the finally block to execute some code regardless of whether an exception occurs or not. This is useful for cleaning up resources or closing files. The syntax of the try-except-finally statement is:\n1try: 2 # code that may cause an exception 3except: 4 # code to run when an exception occurs 5finally: 6 # code to run always Here\u0026rsquo;s how we can modify our genomic example to ensure that the DataFrame is always saved to a file, regardless of whether an exception occurs or not.\n1def process_vcf(df): 2 try: 3 df = filter_by_region(df) 4 df = calculate_vaf(df) 5 calculate_allele_frequency(df, \u0026#34;A\u0026#34;, 1000) 6 # ... more mini functions 7 print(df) 8 except EmptyDataFrameException as e: 9 print(f\u0026#34;Exiting the program due to {e}\u0026#34;) 10 return df 11 finally: 12 df.to_csv(\u0026#34;output.csv\u0026#34;, index=False) 13 print(\u0026#34;The DataFrame was saved to output.csv\u0026#34;) 14 15process_vcf(vcf_df) The complete code can be found in this gist\nConclusion To handle errors due to an empty dataframe, we can use two strategies: check if the DataFrame is non-empty before applying any logic in each mini function and make the pipeline function fail graciously if it receives an empty DataFrame from any of the mini functions by using a custom exception and a try-except block.\n","permalink":"https://firas.phd/posts/python-exception-handling/","summary":"\u003cp\u003eIn genomic data analysis, we often use a pipeline function to process data stored in a dataframe by calling several mini-functions. Each mini-function may modify the dataframe by adding a new column with new values and then filter out the rows that do not meet certain criteria. However, this may result in an empty dataframe if none of the rows satisfy the filters and can lead to errors or unexpected results when the pipeline function tries to perform more operations on the empty dataframe. To avoid this situation, we can use two strategies. First, we can check if the DataFrame is non-empty before applying any logic in each mini-function. Second, we can make the pipeline function fail graciously if it receives an empty DataFrame from any of the mini-functions by using a custom exception and a try-except block. Let\u0026rsquo;s take a look.\u003c/p\u003e","title":"Python Exception Handling"},{"content":"When writing Python scripts, it\u0026rsquo;s common to want to display information messages in the user terminal and possibly save them to a file like output.log. Therefore, it\u0026rsquo;s essential to understand the levels of logging in Python, the types of outputs in Unix-like systems, and how to direct the appropriate type of logging to the right output.\nUnderstanding Output Streams in Unix In Unix systems, there are two primary streams for output: standard output (stdout) and standard error (stderr). These streams serve different purposes:\nStandard Output (stdout): This stream is used by programs to display normal output to the user. It typically includes informative messages, data, and command results. By default, stdout is directed to the terminal, but you can redirect it to files or other processes. Standard Error (stderr): Programs use this stream to display error output to the user, containing error messages and diagnostic information. It provides a separate channel for communicating problems. Like stdout, stderr is directed to the terminal by default, but it can also be redirected. Understanding Python Logging Levels Python\u0026rsquo;s logging module tracks events that occur during program execution, aiding in debugging errors, monitoring performance, and analysing usage patterns. The module includes different levels of severity for logging messages:\nDEBUG: Detailed information, primarily useful for diagnosing problems. INFO: Confirmation that things are working as expected. WARNING: An indication that something unexpected happened, or a problem may arise in the near future. ERROR: Indicates a more serious problem where the software couldn\u0026rsquo;t perform a function. CRITICAL: A severe error, suggesting that the program may be unable to continue running. By default, only messages with a level of WARNING or higher are recorded, but this can be configured. Additionally, logging output can be directed to various destinations, such as files, consoles, or other processes.\nExample Implementation of Logging in Python Let\u0026rsquo;s look at a minimal example of implementing logging in a Python script:\n1#!/usr/bin/env python 2 3import logging 4 5logger = logging.getLogger(\u0026#34;example\u0026#34;) 6logger.setLevel(logging.WARNING) 7 8file_handler = logging.FileHandler(\u0026#34;example.log\u0026#34;) 9logger.addHandler(file_handler) 10 11logger.debug(\u0026#34;This is a debug message\u0026#34;) 12logger.info(\u0026#34;This is an info message\u0026#34;) 13logger.warning(\u0026#34;This is a warning message\u0026#34;) 14logger.error(\u0026#34;This is an error message\u0026#34;) 15logger.critical(\u0026#34;This is a critical message\u0026#34;) By running this script from the command line, a file named example.log is created with the logged messages from the warning level upwards:\n1This is a warning message 2This is an error message 3This is a critical message To also display output in the terminal, we need to add a console handler:\n1console_handler = logging.StreamHandler() 2logger.addHandler(console_handler) Now, logging output will appear in both the terminal and the example.log file.\nHandling Output Redirection in Shell Scripts Suppose we want the script to default to terminal output and then redirect it to example.log using the Unix output redirection operator \u0026gt; operator. In that case, we can structure the script like this:\n1import logging 2 3logger = logging.getLogger(\u0026#34;example\u0026#34;) 4logger.setLevel(logging.WARNING) 5 6console_handler = logging.StreamHandler() 7 8logger.debug(\u0026#34;This is a debug message\u0026#34;) 9logger.info(\u0026#34;This is an info message\u0026#34;) 10logger.warning(\u0026#34;This is a warning message\u0026#34;) 11logger.error(\u0026#34;This is an error message\u0026#34;) 12logger.critical(\u0026#34;This is a critical message\u0026#34;) Running this script from the command line will output to the terminal, but not to any file. To redirect output to an example.log file, we expect that we do so by using the Unix \u0026gt; operator:\n1python log_example.py \u0026gt; example.log However, looking into the content of example.log, we see that the file is empty! So, what is going on?\nDealing with Python Logging Output The confusion mentioned above arises because Python\u0026rsquo;s logging module directs output to stderr by default instead of stdout, whereas the \u0026gt; operator in Unix is employed to redirect stdout rather than stderr. To address this issue, we have several potential solutions.\nRedirecting Both stdout and stderr: Use the \u0026amp;\u0026gt; operator in the shell to redirect both streams to one file. Separating stderr: Use the 2\u0026gt; operator to capture stderr in a separate file while keeping stdout in the terminal. Changing Python Logging: Adjust the Python script to direct logging output to stdout instead of stderr, making it easier to redirect using shell commands. This can be done as follows: 1import sys 2import logging 3 4# Configure logging to direct output to stdout 5console_handler = logging.StreamHandler(sys.stdout) Using verbose for Logging Levels Another useful tip is to pass a verbose argument to determine the level of logging:\n1import sys 2import logging 3 4# Configure logging to direct output to stdout 5console_handler = logging.StreamHandler(sys.stdout) 6console_handler.setLevel(logging.DEBUG if verbose else logging.INFO) This allows flexibility in setting the logging level based on the verbosity of the script\u0026rsquo;s execution.\nConclusion\nUnderstanding how to direct output in both Python and shell scripting can streamline the debugging process. Whether redirecting output in the shell or adjusting Python scripts, mastering these techniques improves our log management skills, making development and troubleshooting more efficient.\n","permalink":"https://firas.phd/posts/python-output-terminal/","summary":"\u003cp\u003eWhen writing Python scripts, it\u0026rsquo;s common to want to display information messages in the user terminal and possibly save them to a file like \u003ccode\u003eoutput.log\u003c/code\u003e. Therefore, it\u0026rsquo;s essential to understand the levels of logging in Python, the types of outputs in Unix-like systems, and how to direct the appropriate type of logging to the right output.\u003c/p\u003e\n\u003ch2 id=\"understanding-output-streams-in-unix\"\u003eUnderstanding Output Streams in Unix\u003c/h2\u003e\n\u003cp\u003eIn Unix systems, there are two primary streams for output: standard output (stdout) and standard error (stderr). These streams serve different purposes:\u003c/p\u003e","title":"Directing Python Output in Terminal"},{"content":"Today, I am installing a Python package on one of the high-performance computing (HPC) systems that I work with. This CentOS 7 setup resembles a fortress, with its stringent security protocols, understandable due to hosting clinical data, but making it quite a challenge to install essential Python packages from public repositories.\nThe package I\u0026rsquo;m attempting to install is PyEnsembl, a handy tool for accessing Ensembl\u0026rsquo;s genetic information.\nPyEnsembl Setup I started by creating a new conda environment:\n1conda create --name ensembl python=3.11 Since conda tends to be sluggish when resolving dependencies, I prefer to install mamba first, which serves as a faster alternative to conda. However, a recent change in conda has made it reasonably faster by incorporating the limbmamba solver (Alternatively, you can consider micromamba):\n1conda install -n base -n defaults \u0026#39;conda\u0026gt;=23.11\u0026#39; After activating the environment, I proceeded to install pyensembl:\n1conda activate ensembl 2conda install pyensembl Genome Setup Before using PyEnsembl, the first step is to download and convert the genome FASTA sequencing files and GTF (Gene Transfer Format) files, which contain genomic annotations, into a structured database format, specifically SQLite. PyEnsembl simplifies this process by parsing the annotation files, generating the database, and providing access to genomic information as Python objects, eliminating the need for direct interaction with the underlying database.\nLet\u0026rsquo;s install the latest release of the human genome \u0026lsquo;GRCh38\u0026rsquo; using the following commands:\n1from pyensembl import EnsemblRelease 2ensembl_38 = EnsemblRelease(release=111, species=\u0026#34;homo_sapiens\u0026#34;) 3ensembl_38.download() 4ensembl_38.index() However, I encountered an SSL-related error:\n1urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in the certificate chain (_ss1.c:1002) This issue stemmed from the SSL configuration in CentOS 7 rather than pyensembl itself. Instead of disabling security as mentioned here, I chose to manually download the required files so that pyensembl could access them from the cache directory.\nFirst, I set up the cache directory:\n1export PYENSEMBL_CACHE_DIR=~/pyensembl-cache Then, I installed the genome using the command line:\n1pyensembl install --species human --release 111 Every time I ran this command, it provided the URL of the required file, which I manually downloaded using wget. In total, I needed four files:\n1wget https://ftp.ensembl.org/pub/release-111/gtf/homo_sapiens/Homo_sapiens.GRCh38.111.gtf.gz 2wget https://ftp.ensembl.org/pub/release-111/fasta/homo_sapiens/cdna/Homo_sapiens.GRCh38.cdna.all.fa.gz 3wget https://ftp.ensembl.org/pub/release-111/fasta/homo_sapiens/ncrna/Homo_sapiens.GRCh38.ncrna.fa.gz 4wget https://ftp.ensembl.org/pub/release-111/fasta/homo_sapiens/pep/Homo_sapiens.GRCh38.pep.all.fa.gz With the files downloaded, I proceeded to install them:\n1from pyensembl import EnsemblRelease 2ensembl_38 = EnsemblRelease(release=111, species=\u0026#34;homo_sapiens\u0026#34;) 3ensembl_38.download() # a redundant step after downloading the files above 4ensembl_38.index() Dependencies Troubleshooting However, I encountered another error:\n1AttributeError: module \u0026#39;polars\u0026#39; has no attribute \u0026#39;enable_string_cache\u0026#39;. Did you mean: \u0026#39;toggle_string_cache\u0026#39;? Further investigation led me to a similar issue on the pyensembl library GitHub repository (issue), which was attributed to a breaking change in the polars library. The installed version of polars was outdated (0.14.28), so I attempted to upgrade to the latest version.\nSince conda was not helpful in upgrading the package from the local outdated offline repository, I chose to download the ``.whl` file from PyPI directly and use pip instead. I typically avoid using pip because it conflicts with conda.\n1# didn\u0026#39;t work 2install -c conda-forge --name ensembl polars 3 4# Download the wheel `.whl` file for `polars` 5wget https://files.pythonhosted.org/packages/2a/b6/89628dbea4624ba0c0c4d960441645b4fa77f830ee3dbbe62bccf4cb0a87/polars-0.20.7-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 6 7# didn\u0026#39;t work 8conda install --no-deps polars-0.20.7-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 9 10# worked 11pip install polars-0.20.7-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl After upgrading polars, I encountered yet another error:\n1AttributeError: module \u0026#39;numpy\u0026#39; has no attribute \u0026#39;typeDict\u0026#39;. Did you mean: \u0026#39;sctypeDict\u0026#39;? A search led me to another issue on the pyensembl GitHub repository (issue), which attributed the problem to the datacache package. The installed version was also outdated (1.1.5).\nOnce again, I had to resort to PyPI and using pip:\n1# Download the wheel `.whl` file for `datacache` 2wget https://files.pythonhosted.org/packages/86/34/606bbcfa507ddc0209f75865e3684a61853925f99b9ffd164d78e2b86271/datacache-1.4.0-py3-none-any.whl 3 4pip install datacache-1.4.0-py3-none-any.whl With both packages upgraded, I tried installing again:\n1pyensembl install --species human --release 111 And finally, it worked as expected. Time to return to my data analysis!\n","permalink":"https://firas.phd/posts/pyensembl_centos_setup/","summary":"\u003cp\u003eToday, I am installing a Python package on one of the high-performance computing (HPC) systems that I work with. This CentOS 7 setup resembles a fortress, with its stringent security protocols, understandable due to hosting clinical data, but making it quite a challenge to install essential Python packages from public repositories.\u003c/p\u003e\n\u003cp\u003eThe package I\u0026rsquo;m attempting to install is \u003ccode\u003ePyEnsembl\u003c/code\u003e, a handy tool for accessing Ensembl\u0026rsquo;s genetic information.\u003c/p\u003e\n\u003ch2 id=\"pyensembl-setup\"\u003ePyEnsembl Setup\u003c/h2\u003e\n\u003cp\u003eI started by creating a new conda environment:\u003c/p\u003e","title":"Running PyEnsembl on CentOS HPC"},{"content":"Debugging, an essential process for identifying and rectifying errors in a computer program, is particularly crucial for computational biologists dealing with complex codebases that often involve intricate mathematical models, data analysis, and simulations. Merely reading the code may not suffice to grasp the logic and functionality of the project. To gain a deeper understanding, you may need to run the code, examine the variables, and observe the outputs. Towards this end, the integrated debuggers in PyCharm and VSCode prove invaluable.\nPyCharm: A Robust and Feature-Rich IDE PyCharm, a dedicated Python IDE, comes in two editions: Professional (paid) and Community (free and open-source). While most debugging features are available in both editions, some advanced features, such as remote debugging, are exclusive to the Professional Edition.\nOne of PyCharm\u0026rsquo;s key strengths lies in its intuitive variable viewer, providing real-time insights into values, types, shapes, and contents of variables. Additionally, the viewer allows for the modification of variables, facilitating the testing of various scenarios.\nInline debugging is another useful feature of PyCharm, displaying variable values next to code lines without the need to hover over them or switch to the debugger window. This saves time and enhances code readability.\nExpressions can be evaluated in the debugger window, editor, or console. The Quick Evaluate Expression feature allows the evaluation of any expression under the caret, while the Evaluate Expression dialog executes arbitrary code in the context of the current frame.\nThe built-in refactoring features are unparalleled, leveraging the intelligence embedded in the IDE to identify opportunities for improving and enhancing code quality and maintenance.\nThe ability to step through the logic of the codebase without inserting breakpoints at every line is an essential feature. I use it consistently whenever I approach an unfamiliar codebase.\nTo add a configuration file in PyCharm, head to the \u0026lsquo;Run/Debug menu\u0026rsquo; and choose \u0026lsquo;Edit Configurations\u0026rsquo;. Here, details such as the main.py script in the package and any arguments can be specified.\nPyCharm also recognises Poetry virtual environments automatically and can install or update packages within the virtual environment as required.\nHowever, PyCharm has its drawback, as it can be resource-intensive, consuming significant memory and CPU power, potentially affecting system performance and battery life (based on my experience with Apple Silicon).\nVSCode: A Lightweight and Cost-Effective IDE VSCode, a versatile and free IDE with a large community, supports multiple languages, including Python. While lacking some advanced features present in PyCharm, VSCode\u0026rsquo;s lightweight design and quick performance make it ideal for small to medium-sized projects.\nTo debug Python code in VSCode, install the Python extension, which provides various features and tools for Python development, including debugging. A launch.json file must be created, containing debugger configurations. This file can be generated by clicking on the \u0026lsquo;Debug icon\u0026rsquo; in the \u0026lsquo;Activity Bar\u0026rsquo;, then clicking on the gear icon to open the \u0026lsquo;Debug Configuration\u0026rsquo; menu. Choose the Python option, and VSCode will generate a default launch.json file, which can be modified according to needs, specifying script path, arguments, environment variables, and console type.\n1{ 2 // Use IntelliSense to learn about possible attributes. 3 // Hover to view descriptions of existing attributes. 4 // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387 5 \u0026#34;version\u0026#34;: \u0026#34;0.2.0\u0026#34;, 6 \u0026#34;configurations\u0026#34;: [ 7 { 8 \u0026#34;name\u0026#34;: \u0026#34;pkgName debug\u0026#34;, 9 \u0026#34;type\u0026#34;: \u0026#34;debugpy\u0026#34;, 10 \u0026#34;request\u0026#34;: \u0026#34;launch\u0026#34;, 11 \u0026#34;program\u0026#34;: \u0026#34;src/pkgName/main.py\u0026#34;, 12 \u0026#34;console\u0026#34;: \u0026#34;integratedTerminal\u0026#34;, 13 \u0026#34;justMyCode\u0026#34;: true, 14 \u0026#34;cwd\u0026#34;: \u0026#34;${workspaceFolder}\u0026#34;, 15 \u0026#34;args\u0026#34;: [ 16 \u0026#34;-i\u0026#34;, 17 \u0026#34;_data/test.input.file\u0026#34;, 18 \u0026#34;-o\u0026#34;, 19 \u0026#34;_data/test.out.file\u0026#34;, 20 \u0026#34;-p\u0026#34;, 21 \u0026#34;8\u0026#34; 22 ] 23 } 24 ] 25} As I use Poetry to manage my virtual Python environments, I\u0026rsquo;ve incorporated the following configuration in the settings.json file to ensure that VSCode recognises them automatically:\n1// to automatically recognises virtualenv interpreters without starting code from an activated shell 2 \u0026#34;python.venvFolders\u0026#34;: [ 3 \u0026#34;~/Library/Caches/pypoetry/virtualenvs\u0026#34; 4 ], 5// to manually specify a default interpreter that will be used when you first open your workspace 6 \u0026#34;python.defaultInterpreterPath\u0026#34;: \u0026#34;${env:PROJ_VENV}/bin/python\u0026#34; However, the variable viewer in VSCode lacks the intuitive and detailed interface of PyCharm. Additionally, it struggles to handle complex nested objects with slots efficiently. On a positive note, the recent introduction of refactoring features in VSCode is a step in the right direction. Unlike PyCharm, these features are compatible with Jupyter Notebook, which is an important aspect to consider. Credit is due to the developers and contributors for providing this tool for free, given that VSCode is a community-driven, open-source project. With the momentum of progress within the VSCode community, I am confident that they will swiftly bridge the gaps.\nAs a side note, the progress of VSCode is certainly raising the bar for PyCharm. This is evident in the recently undertaken, albeit overdue, overhaul of the PyCharm UI and the addition of the integrated LightEdit mode in their product. This mode is particularly advantageous for users who value resource efficiency, especially when they are only viewing code or making simple adjustments.\nConclusion Debugging is an essential skill for any code developer, aiding in understanding and improving code. PyCharm and VSCode are popular IDEs for Python development, each with its advantages and disadvantages. PyCharm offers a comprehensive and user-friendly debugging experience but is resource-intensive and costly. VSCode provides a lightweight and cost-effective debugging experience but lacks some advanced features of PyCharm. Both IDEs can be used for different purposes, such as PyCharm for complex and large-scale projects and VSCode for simple and quick tasks.\n","permalink":"https://firas.phd/posts/python-code-debugging/","summary":"\u003cp\u003eDebugging, an essential process for identifying and rectifying errors in a computer program, is particularly crucial for computational biologists dealing with complex codebases that often involve intricate mathematical models, data analysis, and simulations. Merely reading the code may not suffice to grasp the logic and functionality of the project. To gain a deeper understanding, you may need to run the code, examine the variables, and observe the outputs. Towards this end, the integrated debuggers in PyCharm and VSCode prove invaluable.\u003c/p\u003e","title":"Debugging Python codebases using PyCharm and VSCode"},{"content":"When it comes to Python development on macOS, I rely on a combination of two tools that have served me exceptionally well over the past few years: Pyenv and Poetry. Pyenv provides an elegant solution for managing different Python versions on my system, while Poetry simplifies dependency management and the creation of virtual environments for my projects. In this article, I will guide you through the process of setting up Pyenv to install a specific Python version and then using Poetry to create a virtual environment for your project.\nSetting up Pyenv Install Homebrew: If you are not using Homebrew already, proceed to install it using the following command:\n1/bin/bash -c \u0026#34;$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)\u0026#34; Install Pyenv: Use the following shell command,\n1brew install pyenv and then add the following lines to the shell profile file ~/.zshrc or its equivalent:\n1 # Python configuration ----------------- 2path+=\u0026#34;$HOME/.pyenv/bin\u0026#34; 3if command -v pyenv 1\u0026gt;/dev/null 2\u0026gt;\u0026amp;1; then 4 eval \u0026#34;$(pyenv init -)\u0026#34; 5fi 6alias brew=\u0026#39;env path+=\u0026#34;${PATH//$(pyenv root)\\/shims:/}\u0026#34; brew\u0026#39; The last alias is to ensure that brew works seamlessly with the global Python environment set by pyenv (see here).\nInstall a Specific Python Version: 1pyenv install -l # to list all the available python versions 2env PYTHON_CONFIGURE_OPTS=\u0026#34;--enable-framework\u0026#34; pyenv install 3.11.5 The inclusion of --enable-framework in the install command addresses potential compilation errors during package building (see here).\nSet Global or Local Python Version: Now that pyenv is installed, set the Python version globally or locally. Setting globally provides access to this Python version from anywhere:\n1pyenv global 3.x.x Setting locally provides access to this Python version from a specific directory or project:\n1vim .python-version # add this file to the root of the project 2pyenv local 3.x.x # add this line to the above file Make sure to restart the shell before proceeding.\nUsing Poetry for Dependency Management and Virtual Environments Install Poetry: Install Poetry using the following command:\n1curl -sSL https://install.python-poetry.org | python3 - Initialize Poetry in the Project: 1poetry init Follow the prompts to provide project information, and Poetry will generate a pyproject.toml file.\nInstall Dependencies and Create Virtual Environment: 1poetry install Activate the Virtual Environment: Activate the virtual environment with:\n1poetry install To exit the virtual environment, type exit.\nTo have more control over the structure of the Python project, an alternative approach is to create a project by following these steps:\n1# create a folder for the project 2mkdir ~/Data/Work/Projects/projFolder 3 4# create a virtual env 5poetry new --name pkgName --src . 6 7# install any dependencies 8poetry add biopython 9poetry add numpy 10poetry add pandas 11... The resulting folder structure will look like this:\n1. 2├── README.md 3├── poetry.lock 4├── pyproject.toml 5├── src 6│ └── pkgName 7│ └── __init__.py 8└── tests 9\t└── __init__.py Additionally, I prefer to segregate the dependencies of my Python project into main and dev groups:\n1# add dependencies to the development section 2poetry add --group dev ipykernel black mypy ruff Looking at the pyproject.toml, we can observe two sections: the main section, which comprises the core dependencies of the project, and the development dependencies installed under the dev group.\n1[tool.poetry.dependencies] 2python = \u0026#34;^3.11\u0026#34; 3typer = \u0026#34;^0.9.0\u0026#34; 4bio = \u0026#34;^1.5.9\u0026#34; 5loguru = \u0026#34;^0.7.0\u0026#34; 6plotnine = \u0026#34;^0.12.1\u0026#34; 7pyarrow = \u0026#34;^12.0.0\u0026#34; 8 9[tool.poetry.group.dev.dependencies] 10black = {extras = [\u0026#34;all\u0026#34;], version = \u0026#34;^23.3.0\u0026#34;, allow-prereleases = true} 11jupyter = \u0026#34;^1.0.0\u0026#34; 12mypy = \u0026#34;^1.3.0\u0026#34; 13ruff = \u0026#34;^0.0.270\u0026#34; 14pandas-stubs = \u0026#34;^2.0.1.230501\u0026#34; Updating dependencies To update the project dependencies, we can manually edit the pyproject.toml and run the poetry install command again. Alternatively, we can automate this process using a Poetry plugin called poetry-plugin-up (repo). This plugin updates the version numbers of the dependencies listed in pyproject.toml to their latest compatible versions while respecting version constraints.\n1poetry self add poetry-plugin-up 2poetry up To update all versions to the latest available compatible versions, use the --latest flag.\nMoreover, it is possible to update a specific dependency, only the dependencies listed in the main group, or all of them except the dependencies listed in the dev group.\n1poetry up foo bar 2poetry up --only main 3poetry up --without dev Installing dependencies from repositories In some cases, you may find it necessary to install Python packages directly from repositories, especially when dealing with cutting-edge features, bug fixes, or custom modifications not yet included in official releases. This approach allows you to access the latest developments or specific branches, tag, or revsion of a project.\n1poetry add git+https://git@github.com/has2k1/plotnine.git Or, you can edit the pyproject.toml to include the specific branch, tag, or revision:\n1[tool.poetry.dependencies] 2plotnine = {git = \u0026#34;https://git@github.com/has2k1/plotnine.git\u0026#34;} 3plotnine = {git = \u0026#34;https://git@github.com/has2k1/plotnine.git\u0026#34;, branch = \u0026#34;dev\u0026#34;} 4plotnine = {git = \u0026#34;https://git@github.com/has2k1/plotnine.git\u0026#34;, tag = \u0026#34;v0.12.4\u0026#34;} 5plotnine = {git = \u0026#34;https://git@github.com/has2k1/plotnine.git\u0026#34;, rev = \u0026#34;f78a0fd\u0026#34;} Install packages as editable Installing packages from local folders and in editable mode is another powerful technique that facilitates a dynamic development environment. This mode, often referred to as \u0026ldquo;editable\u0026rdquo; or \u0026ldquo;develop\u0026rdquo; mode, allows you to work on the source code of a package directly within your project. I use it when actively developing or debugging a Python package and needing to test changes in a live environment. Installing the package in editable mode within my project enables me to experiment without the need for repeated installations.\n1poetry add ../my-package/ # install a local package 2poetry add --editable ../my-package/ # install a local package in editable mode 3poetry add ../my-package/dist/my-package-0.1.0.tar.gz 4poetry add ../my-package/dist/my_package-0.1.0.whl Or, you can edit the pyproject.toml manually:\n1[tool.poetry.dependencies] 2my-package = {path = \u0026#34;../my-package/\u0026#34;, develop = true} 3my-package = {path = \u0026#34;../my-package/dist/my-package-0.1.0.tar.gz\u0026#34;} 4my-package = {path = \u0026#34;../my-package/dist/my_package-0.1.0.whl\u0026#34;} Conclusion Pyenv ensures that I have the correct Python version for my projects, while Poetry simplifies dependency management and virtual environment creation. With these tools in my arsenal, Python development becomes efficient, reliable, and reproducible on my system.\n","permalink":"https://firas.phd/posts/python_env_mgmt/","summary":"\u003cp\u003eWhen it comes to Python development on macOS, I rely on a combination of two tools that have served me exceptionally well over the past few years: Pyenv and Poetry. Pyenv provides an elegant solution for managing different Python versions on my system, while Poetry simplifies dependency management and the creation of virtual environments for my projects. In this article, I will guide you through the process of setting up Pyenv to install a specific Python version and then using Poetry to create a virtual environment for your project.\u003c/p\u003e","title":"Managing Python virtual environments"},{"content":"NetMHCPan, a widely used tool for predicting peptide binding to major histocompatibility complex (MHC) molecules, is essential for understanding immune responses. However, the tool\u0026rsquo;s binaries are currently available only for the x86_64 architecture, whether on Darwin (macOS) or Linux. As I intended to conduct test runs on my Apple Silicon device (arm64), I encountered the following error:\n1netMHCpan: no binaries found for Darwin_arm64 /net/sund-nas.win.dtu.dk/storage/services/www/packages/netMHCpan/4.1/netMHCpan-4.1/Darwin_arm64/bin/netMHCpan To address this, one option is to run NetMHCPan using Rosetta, a dynamic binary translator that translates executable code on-the-fly. In simple terms, Rosetta intercepts instructions intended for one architecture (Intel, in this case) and converts them into instructions compatible with another architecture (Apple Silicon).\nEnsure that Rosetta is installed on your machine. If not, use the command softwareupdate --install-rosetta to install it.\nLet\u0026rsquo;s walk through the steps of setting up NetMHCPan on Apple Silicon.\nObtain Binaries and Data Visit the NetMHCPan website and navigate to the Downloads tab. Download either netMHCpan-4.1b.Darwin.tar.gz or netMHCpan-4.1b.Linux.tar.gz, depending on your platform Download data.tar.gz Organise Binaries and Data Extract both the binary and data compressed files into a new folder, for instance, ~/Tools/netMHCpan. Ensure to correctly place the data folder in the designated location.\n1mkdir ~/Tools/netMHCpan 2cd ~/Tools/netMHCpan 3 4tar -xvf netMHCpan-4.1b.Darwin.tar.gz 5tar -xvf data.tar.gz Move the data into the netMHCpan/netMHCpan-4.1 folder.\n1mv data netMHCpan-4.1/ 2cd ~/Tools/netMHCpan/netMHCpan-4.1/Darwin_x86_64/ Here is the tree structure of the ~/Tools/netMHCpan folder:\n1firas@MBP ~/Tools/netMHCpan took 458ms 2024-01-22 20:07:34 2❯ tree -L 3 3. 4└── netMHCpan-4.1 5 ├── Darwin_x86_64 6 │ ├── bin 7 │ └── data -\u0026gt; ../data 8 ├── data 9 │ ├── B0702.fsa 10 │ ├── MHC_pseudo.dat 11 │ ├── README 12 │ ├── all_varcontacts.nlist 13 │ ├── allelenames 14 │ ├── matrices 15 │ ├── synlist 16 │ ├── synlist.bin 17 │ ├── synlist_netMHCpan4.1 18 │ ├── synlist_netMHCpan4.1b 19 │ ├── threshold 20 │ ├── training.pseudo 21 │ └── version 22 ├── netMHCpan 23 ├── netMHCpan-4.1.readme 24 ├── netMHCpan.1 25 └── test 26 ├── B0702.fsa 27 ├── NetMHCpan_out.xls 28 ├── comm 29 ├── test.fsa 30 ├── test.fsa.out 31 ├── test.pep 32 ├── test.pep.out 33 ├── test.pep_BA.out 34 ├── test.pep_userMHC.out 35 └── test.pep_wBA.out Modify for Apple Silicon Now, edit the file ~/Tools/netMHCpan/netMHCpan-4.1/netMHCpan to change the following three lines:\n1setenv NMHOME \u0026#39;~/Tools/netMHCpan/netMHCpan-4.1\u0026#39; 2setenv UNIX \u0026#39;Darwin\u0026#39; 3setenv AR \u0026#39;x86_64\u0026#39; In essence, we are specifying the current location of the netMHCpan folder on our system and then hard-coding the X86_64 architecture by disabling the checks.\nHere is the fully edited version:\n1#! /bin/tcsh -f 2 3# This the main NetMHCpan 4.1 script. It only acts as the frontend to the 4# software proper, a compiled binary. 5# 6# VERSION: 2019 Dec 9 launch 7# 8 9############################################################################### 10# GENERAL SETTINGS: CUSTOMIZE TO YOUR SITE 11############################################################################### 12 13# full path to the NetMHCpan 4.0 directory (mandatory) 14#setenv NMHOME /net/sund-nas.win.dtu.dk/storage/services/www/packages/netMHCpan/4.1/netMHCpan-4.1 15setenv NMHOME \u0026#39;~/Tools/netMHCpan/netMHCpan-4.1\u0026#39; 16 17# determine where to store temporary files (must be writable to all users) 18 19if ( ${?TMPDIR} == 0 ) then 20 setenv TMPDIR /tmp 21endif 22 23# determine platform (do not change this unless you don\u0026#39;t have \u0026#39;uname\u0026#39;!) 24setenv UNIX \u0026#39;Darwin\u0026#39; #`uname -s` 25setenv AR \u0026#39;x86_64\u0026#39; #`uname -m` 26 27############################################################################### 28# NOTHING SHOULD NEED CHANGING BELOW THIS LINE! 29############################################################################### 30 31# other settings 32set PLATFORM = `echo $UNIX $AR | awk \u0026#39;{print $1\u0026#34;_\u0026#34;$2}\u0026#39;` 33setenv NETMHCpan $NMHOME/$PLATFORM 34setenv DTUIBSWWW www 35setenv NetMHCpanWWWPATH /services/NetMHCpan/tmp/ 36setenv NetMHCpanWWWDIR /usr/opt/www/pub/CBS/services/NetMHCpan/tmp 37 38# main ======================================================================== 39if ( -x $NETMHCpan/bin/netMHCpan ) then 40 $NETMHCpan/bin/netMHCpan $* 41else 42 echo netMHCpan: no binaries found for $PLATFORM $NETMHCpan/bin/netMHCpan 43endif 44 45# end of script =============================================================== With these adjustments, NetMHCPan can be seamlessly utilised for analysis on the Apple Silicon architecture.\n","permalink":"https://firas.phd/posts/macos_netmhcpan/","summary":"\u003cp\u003eNetMHCPan, a widely used tool for predicting peptide binding to major histocompatibility complex (MHC) molecules, is essential for understanding immune responses. However, the tool\u0026rsquo;s binaries are currently available only for the x86_64 architecture, whether on Darwin (macOS) or Linux. As I intended to conduct test runs on my Apple Silicon device (arm64), I encountered the following error:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e1\u003c/span\u003e\u003cspan class=\"cl\"\u003enetMHCpan: no binaries found \u003cspan class=\"k\"\u003efor\u003c/span\u003e Darwin_arm64 /net/sund-nas.win.dtu.dk/storage/services/www/packages/netMHCpan/4.1/netMHCpan-4.1/Darwin_arm64/bin/netMHCpan\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eTo address this, one option is to run NetMHCPan using Rosetta, a dynamic binary translator that translates executable code on-the-fly. In simple terms, Rosetta intercepts instructions intended for one architecture (Intel, in this case) and converts them into instructions compatible with another architecture (Apple Silicon).\u003c/p\u003e","title":"Running NetMHCPan on Apple Silicon"},{"content":"Using renv is an excellent choice for maintaining a clean and reproducible R environment on macOS. Here, I will share my experiences and provide a guide on setting up R on macOS. The post is divided into the following sections:\nInstalling system dependencies required for R libraries using Homebrew. Installing R libraries using renv. Saving and restoring R environments using renv. Installing R libraries hosted on private repositories. Building R libraries from source using Makevars. Let\u0026rsquo;s get started!\nInstalling Homebrew 1# install xcode CLI 2xcode-select --install 3# install homebrew 4/bin/bash -c \u0026#34;$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)\u0026#34; Installing system dependencies Create a Brewfile for batch installation of R, RStudio and the required system libraries\n1brew \u0026#34;ccache\u0026#34; # Object-file caching compiler wrapper 2brew \u0026#34;coreutils\u0026#34; # GNU File, Shell, and Text utilities 3brew \u0026#34;gdal\u0026#34; # Geospatial Data Abstraction Library 4brew \u0026#34;geos\u0026#34; # Geometry Engine 5brew \u0026#34;gfortran\u0026#34; # GNU compiler collection 6brew \u0026#34;graphviz\u0026#34; # Graph visualization software from AT\u0026amp;T and Bell Labs 7brew \u0026#34;libgit2\u0026#34; # C library of Git core methods that is re-entrant and linkable 8brew \u0026#34;libjpeg\u0026#34; # Image manipulation library 9brew \u0026#34;libomp\u0026#34; # LLVM\u0026#39;s OpenMP runtime library 10brew \u0026#34;libxml2\u0026#34; # GNOME XML library 11brew \u0026#34;llvm\u0026#34; # Next-gen compiler infrastructure 12brew \u0026#34;openblas\u0026#34; # Optimized BLAS library 13brew \u0026#34;openssl\u0026#34; # OpenSSL GIO module for glib 14brew \u0026#34;poppler\u0026#34; # PDF rendering library (based on the xpdf-3.0 code base) 15brew \u0026#34;proj\u0026#34; # Cartographic Projections Library 16brew \u0026#34;readline\u0026#34; # Library for command-line editing 17brew \u0026#34;unixodbc\u0026#34; # ODBC 3 connectivity for UNIX 18brew \u0026#34;xz\u0026#34; # General-purpose data compression with high compression ratio 19brew \u0026#34;zlib\u0026#34; # General-purpose lossless data-compression library 20brew \u0026#34;r\u0026#34; 21cask \u0026#34;rstudio\u0026#34; Then run the installation using the following brew command,\n1brew bundle install --force --file Brewfile -v Installing R libraries To set up a reproducible R environment, I recommend using renv. Here\u0026rsquo;s an example of how to create a new environment and install the signature.tools.lib library:\nSet the current project folder as the working directory: 1setwd(\u0026#34;/path/to/your/project/folder\u0026#34;) Install the renv library if you haven\u0026rsquo;t already: 1install.packages(\u0026#34;renv\u0026#34;) Initialize and activate the renv environment: 1library(renv) 2renv::init() 3renv::activate() By using renv, only the renv library itself will be installed in the system-wide location, such as /usr/local/Cellar/r/ (Intel macOS) or /opt/homebrew/Cellar/r/ (Apple Silicon).\nProject-specific libraries, on the other hand, will be installed into the renv base located at ~/Library/Caches/org.R-project.R. When creating a new environment, renv links the libraries from its base into the renv folder within the project directory. This approach maintains a clean system base and allows for sharing the renv space across different R environments.\nTo install project-specific R libraries, you can utilize the renv::install() function. For instance, if you want to install signature.tools.lib, it has the following dependencies:\n1renv::install(\u0026#34;bioc::VariantAnnotation\u0026#34;) 2renv::install(\u0026#34;bioc::BSgenome.Hsapiens.UCSC.hg38\u0026#34;) 3renv::install(\u0026#34;bioc::BSgenome.Hsapiens.1000genomes.hs37d5\u0026#34;) 4renv::install(\u0026#34;bioc::BSgenome.Mmusculus.UCSC.mm10\u0026#34;) 5renv::install(\u0026#34;bioc::BSgenome.Cfamiliaris.UCSC.canFam3\u0026#34;) 6renv::install(\u0026#34;bioc::SummarizedExperiment\u0026#34;) 7renv::install(\u0026#34;bioc::BiocGenerics\u0026#34;) 8renv::install(\u0026#34;bioc::GenomeInfoDb\u0026#34;) 9renv::install(\u0026#34;NMF\u0026#34;) 10renv::install(\u0026#34;foreach\u0026#34;) 11renv::install(\u0026#34;doParallel\u0026#34;) 12renv::install(\u0026#34;lpSolve\u0026#34;) 13renv::install(\u0026#34;ggplot2\u0026#34;) 14renv::install(\u0026#34;cluster\u0026#34;) 15renv::install(\u0026#34;methods\u0026#34;) 16renv::install(\u0026#34;stats\u0026#34;) 17renv::install(\u0026#34;linxihui/NNLM\u0026#34;) 18renv::install(\u0026#34;nnls\u0026#34;) 19renv::install(\u0026#34;GenSA\u0026#34;) 20renv::install(\u0026#34;gmp\u0026#34;) 21renv::install(\u0026#34;plyr\u0026#34;) 22renv::install(\u0026#34;RCircos\u0026#34;) 23renv::install(\u0026#34;scales\u0026#34;) 24renv::install(\u0026#34;bioc::GenomicRanges\u0026#34;) 25renv::install(\u0026#34;bioc::IRanges\u0026#34;) 26renv::install(\u0026#34;bioc::BSgenome\u0026#34;) 27renv::install(\u0026#34;readr\u0026#34;) 28renv::install(\u0026#34;doRNG\u0026#34;) 29renv::install(\u0026#34;combinat\u0026#34;) 30renv::install(\u0026#34;Nik-Zainal-Group/signature.tools.lib.dev\u0026#34;) 31renv::install(\u0026#34;Nik-Zainal-Group/teachingmutationalsignatures\u0026#34;) Saving and restoring R environments To instruct renv to save the environment you described, you can use the renv::snapshot() function. However, please note that renv only saves libraries that are explicitly called in your code. If a library is not referenced anywhere in your code, renv will not include it in the snapshot.\nTo ensure all the required libraries are saved, you can create a separate file, let\u0026rsquo;s call it libraries.R, where you call and load all the necessary libraries. This way, renv will detect these library calls and include them in the snapshot.\nHere\u0026rsquo;s an example of how you can structure the libraries.R file:\n1# libraries.R 2library(signature.tools.lib) 3# Call and load any other required libraries here By calling renv::snapshot() after sourcing the libraries.R file, renv will recognize the library calls and include the required libraries in the snapshot.\n1# save renv.lock file 2renv::snapshot() Saving the environment will generate a renv.lock.\nTo restore the environment on a new location or machine, you can follow these steps after copying the libraries.R and renv.lock files into a new project folder, then run the following commands:\n1# if `renv` is not installed on the new system 2install.packages(\u0026#39;renv\u0026#39;) 3 4renv::init() 5# or 6renv::restore() Installation from private repositories When dealing with R libraries hosted on a private repository, additional authentication is required. One common method is to generate a Personal Access Token (PAT) to authenticate with the hosting service like Github.\nTo authenticate using the generated PAT.\n1Sys.setenv(GITHUB_PAT = \u0026#34;PASTE_TOKEN_HERE\u0026#34;) 2renv::install(\u0026#34;private_repo/R_package\u0026#34;) Installation from source To install R libraries from source, we need to instruct R where to find the required system dependencies. This is done by using a Makevars file. which is placed ~/.R/Makevars and looks like this.\n1# -------- 2# Makevars 3# -------- 4 5# General note 6 7# Homebrew bin / opt / lib locations 8 9### uncomment on Apple Silicon 10#HB=/opt/homebrew/bin 11#HO=/opt/homebrew/opt 12#HL=/opt/homebrew/lib 13#HI=/opt/homebrew/include 14 15### uncomment on Intel 16#HB=/usr/local/bin 17#HO=/usr/local/opt 18#HL=/usr/local/lib 19#HI=/usr/local/include 20 21# MacOS Xcode header location 22# (do \u0026#34;xcrun -show-sdk-path\u0026#34; in terminal to get path) 23XH=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk 24 25# ccache 26CCACHE=$(HB)/ccache 27 28# Make using all cores (set # to # of cores on your machine) 29MAKE=make -j4 30 31# GNU version 32GNU_VER=12 33 34# LLVM (Clang) compiler options 35CC=$(CCACHE) $(HO)/llvm/bin/clang 36CXX=$(CC)++ 37CXX98=$(CC)++ 38CXX11=$(CC)++ 39CXX14=$(CC)++ 40CXX17=$(CC)++ 41 42# FORTRAN 43FC=$(CCACHE) $(HB)/gfortran-$(GNU_VER) 44F77=$(FC) 45FLIBS=-L$(HL)/gcc/$(GNU_VER) -lgfortran -lquadmath -lm 46 47# STD libraries 48CXX1XSTD=-std=c++0x 49CXX11STD=-std=c++11 50CXX14STD=-std=c++14 51CXX17STD=-std=c++17 52 53# FLAGS 54STD_FLAGS=-g -O3 -Wall -pedantic -mtune=native -pipe 55CFLAGS=$(STD_FLAGS) 56CXXFLAGS=$(STD_FLAGS) 57CXX98FLAGS=$(STD_FLAGS) 58CXX11FLAGS=$(STD_FLAGS) 59CXX14FLAGS=$(STD_FLAGS) 60CXX17FLAGS=$(STD_FLAGS) 61 62# Preprocessor FLAGS 63# NB: -isysroot refigures the include path to the Xcode SDK we set above 64CPPFLAGS=-isysroot $(XH) -I$(HI) \\ 65 -I$(HO)/llvm/include -I$(HO)/openssl/include \\ 66 -I$(HO)/gettext/include -I$(HO)/tcl-tk/include 67 68# Linker flags (suggested by homebrew) 69LDFLAGS+=-L$(HO)/llvm/lib -Wl,-rpath,$(HO)/llvm/lib 70 71# Flags for OpenMP support that should allow packages that want to use 72# OpenMP to do so (data.table), and other packages that bork with 73# -fopenmp flag (stringi) to be left alone 74SHLIB_OPENMP_CFLAGS=-fopenmp 75SHLIB_OPENMP_CXXFLAGS=-fopenmp 76SHLIB_OPENMP_CXX98FLAGS=-fopenmp 77SHLIB_OPENMP_CXX11FLAGS=-fopenmp 78SHLIB_OPENMP_CXX14FLAGS=-fopenmp 79SHLIB_OPENMP_CXX17FLAGS=-fopenmp 80SHLIB_OPENMP_FCFLAGS=-fopenmp 81SHLIB_OPENMP_FFLAGS=-fopenmp There are three things we need to pay attention to in the file above (highlighted lines):\nDepending on the architecture of the macOS machine, Intel vs Apple Silicon (M1/2), we need to uncomment the corresponding lines. Run xcrun -show-sdk-path to make sure they are located at /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk To find the installed version of GCC, change directory into /usr/local/lib/gcc/ on an Intel macOS, or /opt/homebrew/lib/ on an Apple Silicon. The mentioned method is useful for installing libraries like the data.table library that require OpenMP support to leverage multiprocessing capabilities. For more details on this approach, you can refer to the article located here.\n","permalink":"https://firas.phd/posts/r_macos/","summary":"\u003cp\u003eUsing \u003ccode\u003erenv\u003c/code\u003e is an excellent choice for maintaining a clean and reproducible R environment on macOS. Here, I will share my experiences and provide a guide on setting up R on macOS. The post is divided into the following sections:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003eInstalling system dependencies required for R libraries using Homebrew.\u003c/li\u003e\n\u003cli\u003eInstalling R libraries using \u003ccode\u003erenv\u003c/code\u003e.\u003c/li\u003e\n\u003cli\u003eSaving and restoring R environments using \u003ccode\u003erenv\u003c/code\u003e.\u003c/li\u003e\n\u003cli\u003eInstalling R libraries hosted on private repositories.\u003c/li\u003e\n\u003cli\u003eBuilding R libraries from source using \u003ccode\u003eMakevars\u003c/code\u003e.\u003c/li\u003e\n\u003c/ol\u003e\n\u003cp\u003eLet\u0026rsquo;s get started!\u003c/p\u003e","title":"Setting up a reproducible R environment on macOS"},{"content":"Processing Genomics workflows require a variety of tools with distinct computing resources requirements. Some of these requirements are merely beyond the configuration of powerful workstations. High-performance clusters (HPC) offer the possibility to scale up computing resources to meet the increasing demands of these processes. However, maintaining the infrastructure of an on-premises HPC may not be a viable solution for many parities. A more cost-effective method is to provide computer resources dynamically by leveraging cloud computing. This strategy, combined with a powerful workflow manager such as NextFlow, allow researchers to execute their workflows locally and/or in the cloud with minimal friction.\nAWS Batch One of the available options to build a cloud-based HPC solution is AWS Batch from Amazon. It is a free service that can run batch jobs either periodically or on-demand in a usage-based pricing model.\nAWS Batch can be broken down into the following components:\nCompute Environment defines computing resources for a specific workload. Job Queues allows binding of a specific task to one / more Compute Environment(s). Job Definition a template for one / more job(s) in the workload. Jobs binds a Job Definition to a specific Job Queue. NextFlow supports submitting tasks (Job Definition) to AWS Batch via API calls. AWS Batch, in turns, maps each task with its resources requirements and store them as Jobs in the Job Queue awaiting for computing resources to become available.\nEach Job Queue can be linked to one or more Compute Environnement that, with priority scheduling, distribute tasks of higher priorities to be executed on more than one Compute Environments.\nA Compute Environnement is responsible for dynamically launch and scale compute resources based on tasks recruitments. It can be configured as an on-demand or spot Compute Environnement. In the on-demand model, there is no delay in launching and executing jobs. In contrast, in the spot model, the Compute Environnement is configured to launch Spot instances (i.e. price-reduced EC2 instances) once the Spot price drops below a specific percentage of the on-demand price.\nOnce the compute resources become available, the Amazon Elastic Container Service (ECS) launches EC2 instances with a pre-built custom Amazon Machine Image (AMI) along with the pre-specified Amazon Elastic Block Store (EBS). In addition, it requests any named container from the Amazon Elastic Container Registry (ECR).\nThe EC2 mounts the S3 buckets to read the data, perform the process, and store the results back on S3. Depending on the volume of the data, the transfer between the S3 storage and the EBS may add additional processing time. This can be avoided by using Amazon FSx for Lustre which mounts the file system directly on the EC2 instance.\nThe Setup Before running NextFlow on AWS Batch, we need to enable the service in our AWS account.\nIAM user This user account is used to manage the AWS Batch service. It needs the following privileges to be added within the IAM section:\nSchedule jobs using AWSBatch (AWSBatchFullAccess policy). Access S3 storage (AmazonS3FullAccess policy) Launch/terminate EC2 containers (AmazonEC2FullAccess policy) IAM Services Roles In addition to the account above, the AWS Batch service needs to be configured with the following roles:\nLaunching/terminating on-demand EC2 instances and access S3 storage: AWSBatchServiceRole role with AWSBatchServiceRole permission ecsInstanceRole role with both AmazonS3FullAccess and AmazonEC2ContainerServiceforEC2Role permissions Grant the Spot Fleet permission to launch/terminate EC2 instances: AmazonEC2SpotFleetRole role with both AmazonEC2SpotFleetTaggingRole and AmazonEC2SpotFleetAutoscaleRole policies AWSServiceRoleForEC2SpotFleet role with AWSEC2SpotFleetServiceRolePolicy policy AWS Batch custom AMI (AB AMI) This AMI serves as a template for every launched EC2 instance by AWS Batch. While it is possible to use many different instant types (see here), it is recommended to limit our choices to the ones that are compatible with ECS. The exact processor number and memory capacity are not significant at this stage as they are overridden by the Compute Environment at run-time. However, the attached EBS storage needs to be large enough to handle: the docker image, any genomic indices, input files, temp files and output files for any given process.\nOnce the AMI has started, we need to install awscli to enable read/write access to S3 storage. In addition, we need to increase the size of the Docker storage to match the size of the docker images to be used in the pipeline (see here). Finally, we need to create an image of this AMI to serve as a template for the ec2 instances launched by AWS Batch.\nCompute Environment When creating a Compute Environment, there are two options to consider: 1) the \u0026lsquo;Allowed instance types\u0026rsquo;: dictated by the hard requirements of our processes and 2) the \u0026lsquo;Provisioning model\u0026rsquo; as \u0026lsquo;On-Demand\u0026rsquo; vs \u0026lsquo;Spot\u0026rsquo;: dictated by our budget. In addition, we may specify the minimum and the maximum number of vCPUs. The maximum number of vCPUs is the number of all vCPUs allowed to be launched by the workflow. To get the number of instances that will launch of a given instance type, divide the maximum number by the number of vCPUs of that instance type. The minimum number of vCPUs is what the Compute Environment will maintain at all time, so setting it to zero helps avoid additional costs.\nJob Queue Each Job Queue can be connected to one or more Compute Environments. In addition, specifying the \u0026lsquo;Priority\u0026rsquo; enables the distribution of higher priority jobs to multiple Compute Environment.\nNextFlow AMI (NF AMI) Although Nextflow supports scheduling jobs on the AWS Batch while running locally, it is not ideal for large workflows as it requires a constant connection for the duration of the workflow which could be hours to days. One possibility is to create a NextFlow job submission EC2 instance with a persistent EBS. This instance can be configured with NextFlow along with other productivity tools such as aws-cli, git, tmux, neovim, etc.\nNextFlow Configuration The final step of this setup is to configure NextFlow to work with AWS Batch. For this, we need to create a profile where the executor is awsbatch. In addition, we need to configure the options for queue, aws regions, access credentials, and the container to be used in each of the submitted processes.\nFinal remarks This post provided an overview of how AWS Batch works and how to leverage it for running NextFlow pipelines in the cloud. It reflects my personal experience using the AWS Batch to run Genomic NextFlow pipelines in the cloud. For specific step-by-step guide, one can check the AWS Batch User Guide, and the NextFlow documentation. For reference, my configuration can be found in my dedicated Github repository here.\n","permalink":"https://firas.phd/posts/nextflow_aws/","summary":"\u003cp\u003eProcessing Genomics workflows require a variety of tools with distinct computing resources requirements. Some of these requirements are merely beyond the configuration of powerful workstations. High-performance clusters (HPC) offer the possibility to scale up computing resources to meet the increasing demands of these processes. However, maintaining the infrastructure of an on-premises HPC may not be a viable solution for many parities. A more cost-effective method is to provide computer resources dynamically by leveraging cloud computing. This strategy, combined with a powerful workflow manager such as NextFlow, allow researchers to execute their workflows locally and/or in the cloud with minimal friction.\u003c/p\u003e","title":"Running NextFlow using AWS Batch"},{"content":"When developing in Python, it is generally a good practice not to rely on the Python version that ships with the operating system (OS). This is to ensure that the system version of Python remains relatively \u0026lsquo;clean\u0026rsquo; for the OS processes. In addition, by installing a custom version(s) of Python, we open many possibilities. For one, it gives us control over which specific version(s) to use in our projects, and two, by using a virtual environment manager, we ensure that each project has access to its own tailored list of packages. One way of achieving this is by using pyenv.\nHow does pyenv works? The way pyenv works is that every time we issue a Python command, it intercepts this call, using shim executables injected in our environment\u0026rsquo;s PATH, and redirects the call to the correct Python installation based on our configurations.\nHow to install pyenv? Personally, on macOS, I prefer to use HomeBrew.\n1brew install pyenv pyenv-virtualenv This command will install both pyenv and pyenv-virtualenv which is a virtual environment manager for Python.\nWe also need to add pyenv to our shell ~/.zshrc to enable shims and autocompletion:\n1# Pyenv configruation ---------------- 2if command -v pyenv 1\u0026gt;/dev/null 2\u0026gt;\u0026amp;1; then 3 eval \u0026#34;$(pyenv init -)\u0026#34; 4fi 5if command -v pyenv virtualenv-init 1\u0026gt;/dev/null 2\u0026gt;\u0026amp;1; then 6 eval \u0026#34;$(pyenv virtualenv-init -)\u0026#34; 7fi Once added, restart your terminal or execute exec zsh to reload the above configuration.\nOnce pyenv is installed, we can list all the available versions of Python (there are many!) by issuing the following command.\n1pyenv install --list 2# Available versions: 3# 2.1.3 4# 2.2.3 5# 2.3.7 6# 2.4.0 7# 2.4.1 8# 2.4.2 9# ... output truncated ... To install one of these versions (for example 3.9.4), we would normally use\n1pyenv install 3.9.4 However, to use the environment with reticulate package in RStudio, we need to use env PYTHON_CONFIGURE_OPTS=\u0026quot;--enable-shared\u0026quot; with the above command, otherwise we may receive the following error when loading reticulate in R:\n1Error: Python shared library not found, Python bindings not loaded. Use reticulate::install_miniconda() if you\u0026#39;d like to install a Miniconda Python environment. 21. stop(paste(msg, hint, sep = \u0026#34;\\n\u0026#34;), call. = FALSE) 32. python_not_found(\u0026#34;Python shared library not found, Python bindings not loaded.\u0026#34;) 43. initialize_python(required_module, use_environment) 54. ensure_python_initialized() 65. reticulate::py_config() This is to satisfy one of the requirements of reticulate as specified here.\nNote that for reticulate to bind to a version of Python it must be compiled with shared library support (i.e. with the \u0026ndash;enable-shared flag).\nTo proceed with installing Python 3.9.4 use:\n1env PYTHON_CONFIGURE_OPTS=\u0026#34;--enable-shared\u0026#34; pyenv install 3.9.4 How to prepare a Python environment for reticulate? First, create an environment named, for example, PyRStudio that is based on the Python version we installed above (3.9.4).\n1pyenv virtualenv 3.9.4 PyRStudio Then use pyenv activate PyRStudio to manually activate the environment, or if you prefer to activate this environment automatically every time we change into the project directory, do:\n1cd ~/path/to/projectDir 2pyenv local PyRStudio One last step is to install numpy in the activated environment.\n1pyenv activate PyRStudio 2pip3 install numpy Final check Launch a new Rstudio session, e.g. open -a Rstudio and check whether reticulate can recognise the activated environment.\n1reticulate::py_config() 2# python: /Users/firas/.pyenv/shims/python3 3# libpython: /Users/firas/.pyenv/versions/3.9.4/lib/libpython3.9.dylib 4# pythonhome: /Users/firas/.pyenv/versions/PyRStudio:/Users/firas/.pyenv/versions/PyRStudio 5# version: 3.9.4 (default, Apr 27 2021, 20:14:35) [Clang 12.0.0 (clang-1200.0.32.29)] 6# numpy: /Users/firas/.pyenv/versions/3.9.4/envs/PyRStudio/lib/python3.9/site-packages/numpy 7# numpy_version: 1.20.2 8# 9# python versions found: 10# /Users/firas/.pyenv/shims/python3 11# /usr/bin/python3 12# /usr/local/bin/python3 13# /usr/bin/python And test numpy in a new Python code chunk:\n1import numpy as np 2a = np.array([1, 2, 3]) 3print(a) 4 5# Python 3.9.4 (/Users/firas/.pyenv/shims/python3) 6# Reticulate 1.19 REPL -- A Python interpreter in R. 7# [1 2 3] How to revert this setup? To uninstall the virtual environment, use: 1pyenv uninstall PyRstudio To uninstall both the virtual environment and the custom Python version (3.9.4 in our example), use: 1pyenv uninstall 3.9.4 To uninstall pyenv 1rm -rf $(pyenv root) 2brew uninstall pyenv Additionally, delete/comment the pyenv configuration commands that we added to ~/.zshrc during the installation step above.\n","permalink":"https://firas.phd/posts/pyenv_rstudio/","summary":"\u003cp\u003eWhen developing in Python, it is generally a good practice not to rely on the Python version that ships with the operating system (OS). This is to ensure that the system version of Python remains relatively \u0026lsquo;clean\u0026rsquo; for the OS processes. In addition, by installing a custom version(s) of Python, we open many possibilities. For one, it gives us control over which specific version(s) to use in our projects, and two, by using a virtual environment manager, we ensure that each project has access to its own tailored list of packages. One way of achieving this is by using \u003ca href=\"https://github.com/pyenv/pyenv\"\u003epyenv\u003c/a\u003e.\u003c/p\u003e","title":"Using reticulate in RStudio with pyenv"},{"content":"If you are facing difficulties with large data sets in R, using data.table could provide a performance boost. However, when loading data.table, especially on macOS, you might encounter a warning indicating the absence of OpenMP support, causing data.table to operate in a single-threaded mode. This limitation prevents you from fully utilizing the potential benefits of using data.table and taking advantage of the underlying hardware.\n1library(data.table) 2data.table 1.14.0 using 1 threads (see ?getDTthreads). Latest news: r-datatable.com 3********** 4This installation of data.table has not detected OpenMP support. 5It should still work but in single-threaded mode. 6If this is a Mac, please ensure you are using R\u0026gt;=3.4.0 and have followed our Mac instructions here: 7https://github.com/Rdatatable/data.table/wiki/Installation. 8This warning message should not occur on Windows or Linux. If it does, please file a GitHub issue. 9********** 10 11getDTthreads() 12[1] 1 What is the issue here? OpenMP is an implementation of multithreading, and the Clang compiler that comes with Xcode on macOS lacks support for OpenMP. Apple has chosen not to include the libomp.dylib run-time library in their compiler. You can verify this by executing the following command.\n1$ clang -c omp.c -fopenmp 2clang: error: unsupported option \u0026#39;-fopenmp\u0026#39; To restore support for OpenMP in clang, one way is to 1) install the latest official LLVM release and 2) instruct R to compile data.table with OpenMP support using a Makevars file. Other R packages which support OpenMP will also benefit from this upgrade.\nHow to proceed? Install Homebrew In macOS, It is highly recommended to use HomeBrew package manager to ensure both automation and reproducibility. First, we need to make sure that we have the latest version of Xcode installed.\n1xcode-select --install Then we proceed to install Homebrew using the following command (for more details check the HomeBrew website).\n1/bin/bash -c \u0026#34;$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)\u0026#34; Install LLVM Now, it\u0026rsquo;s time to install the latest version of the LLVM compiler.\n1brew update \u0026amp;\u0026amp; brew install llvm The LLVM compiler installation is keg-only, which means it will shadow the system\u0026rsquo;s compiler instead of overriding it, according to HomeBrew\u0026rsquo;s terminology. In other words, it will not symlink its binaries into /usr/local/. To instruct R to build new packages using the newly installed compiler, you need to create a Makevars file and put it in ~/.R/Makevars.\nEdit Makevars Let\u0026rsquo;s edit the Makevars file to include the paths of the new compiler and the required compiler options to support OpenMP. For example, on my machine using macOS Big Sur Version 11.2.3 (at the time of writing this post), my Makevars file looks as followed.\n1# -------- 2# Makevars 3# -------- 4 5# General note 6 7# Homebrew bin / opt / lib locations 8 9### uncomment on Apple Silicon 10#HB=/opt/homebrew/bin 11#HO=/opt/homebrew/opt 12#HL=/opt/homebrew/lib 13#HI=/opt/homebrew/include 14 15### uncomment on Intel 16#HB=/usr/local/bin 17#HO=/usr/local/opt 18#HL=/usr/local/lib 19#HI=/usr/local/include 20 21# MacOS Xcode header location 22# (do \u0026#34;xcrun -show-sdk-path\u0026#34; in terminal to get path) 23XH=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk 24 25# ccache 26CCACHE=$(HB)/ccache 27 28# Make using all cores (set # to # of cores on your machine) 29MAKE=make -j4 30 31# GNU version 32GNU_VER=12 33 34# LLVM (Clang) compiler options 35CC=$(CCACHE) $(HO)/llvm/bin/clang 36CXX=$(CC)++ 37CXX98=$(CC)++ 38CXX11=$(CC)++ 39CXX14=$(CC)++ 40CXX17=$(CC)++ 41 42# FORTRAN 43FC=$(CCACHE) $(HB)/gfortran-$(GNU_VER) 44F77=$(FC) 45FLIBS=-L$(HL)/gcc/$(GNU_VER) -lgfortran -lquadmath -lm 46 47# STD libraries 48CXX1XSTD=-std=c++0x 49CXX11STD=-std=c++11 50CXX14STD=-std=c++14 51CXX17STD=-std=c++17 52 53# FLAGS 54STD_FLAGS=-g -O3 -Wall -pedantic -mtune=native -pipe 55CFLAGS=$(STD_FLAGS) 56CXXFLAGS=$(STD_FLAGS) 57CXX98FLAGS=$(STD_FLAGS) 58CXX11FLAGS=$(STD_FLAGS) 59CXX14FLAGS=$(STD_FLAGS) 60CXX17FLAGS=$(STD_FLAGS) 61 62# Preprocessor FLAGS 63# NB: -isysroot refigures the include path to the Xcode SDK we set above 64CPPFLAGS=-isysroot $(XH) -I$(HI) \\ 65 -I$(HO)/llvm/include -I$(HO)/openssl/include \\ 66 -I$(HO)/gettext/include -I$(HO)/tcl-tk/include 67 68# Linker flags (suggested by homebrew) 69LDFLAGS+=-L$(HO)/llvm/lib -Wl,-rpath,$(HO)/llvm/lib 70 71# Flags for OpenMP support that should allow packages that want to use 72# OpenMP to do so (data.table), and other packages that bork with 73# -fopenmp flag (stringi) to be left alone 74SHLIB_OPENMP_CFLAGS=-fopenmp 75SHLIB_OPENMP_CXXFLAGS=-fopenmp 76SHLIB_OPENMP_CXX98FLAGS=-fopenmp 77SHLIB_OPENMP_CXX11FLAGS=-fopenmp 78SHLIB_OPENMP_CXX14FLAGS=-fopenmp 79SHLIB_OPENMP_CXX17FLAGS=-fopenmp 80SHLIB_OPENMP_FCFLAGS=-fopenmp 81SHLIB_OPENMP_FFLAGS=-fopenmp There are three things we need to pay attention to in the file above (highlighted lines):\nDepending on the architecture of the macOS machine, Intel vs Apple Silicon (M1/2), we need to uncomment the corresponding lines. Run xcrun -show-sdk-path to make sure they are located at /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk To find the installed version of GCC, change directory into /usr/local/lib/gcc/ on an Intel macOS, or /opt/homebrew/lib/ on an Apple Silicon. Now, with the above Makevars in place. Let\u0026rsquo;s start a new R session and compile data.table from source.\n1remove.packages(\u0026#34;data.table\u0026#34;) 2install.packages(\u0026#34;data.table\u0026#34;, type = \u0026#34;source\u0026#34;, 3 repos = \u0026#34;https://Rdatatable.gitlab.io/data.table\u0026#34;) To check if we have successfully compiled data.table with OpenMP support, let\u0026rsquo;s load the library.\n1library(data.table) 2# data.table 1.14.0 using 4 threads (see ?getDTthreads). Latest news: r-datatable.com The message above shows that our data.table is working in multithreaded mode and is currently using 4 threads. To change the number of threads used by data.table and to make it persistent for every session, edit the ~/.Rprofile and include the following command.\n1# data.table configuration 2data.table::setDTthreads(4) Refs: The Makevars configuration was adapted from the following gist.\n","permalink":"https://firas.phd/posts/data_table_openmp/","summary":"\u003cp\u003eIf you are facing difficulties with large data sets in R, using \u003ccode\u003edata.table\u003c/code\u003e could provide a performance boost. However, when loading \u003ccode\u003edata.table\u003c/code\u003e, especially on macOS, you might encounter a warning indicating the absence of OpenMP support, causing \u003ccode\u003edata.table\u003c/code\u003e to operate in a single-threaded mode. This limitation prevents you from fully utilizing the potential benefits of using \u003ccode\u003edata.table\u003c/code\u003e and taking advantage of the underlying hardware.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-r\" data-lang=\"r\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 1\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nf\"\u003elibrary\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003edata.table\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 2\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003edata.table\u003c/span\u003e \u003cspan class=\"m\"\u003e1.14.0\u003c/span\u003e \u003cspan class=\"n\"\u003eusing\u003c/span\u003e \u003cspan class=\"m\"\u003e1\u003c/span\u003e \u003cspan class=\"nf\"\u003ethreads \u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003esee\u003c/span\u003e \u003cspan class=\"o\"\u003e?\u003c/span\u003e\u003cspan class=\"n\"\u003egetDTthreads\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"n\"\u003e. \u003c/span\u003e \u003cspan class=\"n\"\u003eLatest\u003c/span\u003e \u003cspan class=\"n\"\u003enews\u003c/span\u003e\u003cspan class=\"o\"\u003e:\u003c/span\u003e \u003cspan class=\"n\"\u003er\u003c/span\u003e\u003cspan class=\"o\"\u003e-\u003c/span\u003e\u003cspan class=\"n\"\u003edatatable.com\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 3\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"o\"\u003e**********\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 4\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003eThis\u003c/span\u003e \u003cspan class=\"n\"\u003einstallation\u003c/span\u003e \u003cspan class=\"n\"\u003eof\u003c/span\u003e \u003cspan class=\"n\"\u003edata.table\u003c/span\u003e \u003cspan class=\"n\"\u003ehas\u003c/span\u003e \u003cspan class=\"n\"\u003enot\u003c/span\u003e \u003cspan class=\"n\"\u003edetected\u003c/span\u003e \u003cspan class=\"n\"\u003eOpenMP\u003c/span\u003e \u003cspan class=\"n\"\u003esupport.\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 5\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003eIt\u003c/span\u003e \u003cspan class=\"n\"\u003eshould\u003c/span\u003e \u003cspan class=\"n\"\u003estill\u003c/span\u003e \u003cspan class=\"n\"\u003ework\u003c/span\u003e \u003cspan class=\"n\"\u003ebut\u003c/span\u003e \u003cspan class=\"kr\"\u003ein\u003c/span\u003e \u003cspan class=\"n\"\u003esingle\u003c/span\u003e\u003cspan class=\"o\"\u003e-\u003c/span\u003e\u003cspan class=\"n\"\u003ethreaded\u003c/span\u003e \u003cspan class=\"n\"\u003emode.\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 6\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003eIf\u003c/span\u003e \u003cspan class=\"n\"\u003ethis\u003c/span\u003e \u003cspan class=\"n\"\u003eis\u003c/span\u003e \u003cspan class=\"n\"\u003ea\u003c/span\u003e \u003cspan class=\"n\"\u003eMac\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"n\"\u003eplease\u003c/span\u003e \u003cspan class=\"n\"\u003eensure\u003c/span\u003e \u003cspan class=\"n\"\u003eyou\u003c/span\u003e \u003cspan class=\"n\"\u003eare\u003c/span\u003e \u003cspan class=\"n\"\u003eusing\u003c/span\u003e \u003cspan class=\"n\"\u003eR\u003c/span\u003e\u003cspan class=\"o\"\u003e\u0026gt;=\u003c/span\u003e\u003cspan class=\"m\"\u003e3.4.0\u003c/span\u003e \u003cspan class=\"n\"\u003eand\u003c/span\u003e \u003cspan class=\"n\"\u003ehave\u003c/span\u003e \u003cspan class=\"n\"\u003efollowed\u003c/span\u003e \u003cspan class=\"n\"\u003eour\u003c/span\u003e \u003cspan class=\"n\"\u003eMac\u003c/span\u003e \u003cspan class=\"n\"\u003einstructions\u003c/span\u003e \u003cspan class=\"n\"\u003ehere\u003c/span\u003e\u003cspan class=\"o\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 7\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003ehttps\u003c/span\u003e\u003cspan class=\"o\"\u003e://\u003c/span\u003e\u003cspan class=\"n\"\u003egithub.com\u003c/span\u003e\u003cspan class=\"o\"\u003e/\u003c/span\u003e\u003cspan class=\"n\"\u003eRdatatable\u003c/span\u003e\u003cspan class=\"o\"\u003e/\u003c/span\u003e\u003cspan class=\"n\"\u003edata.table\u003c/span\u003e\u003cspan class=\"o\"\u003e/\u003c/span\u003e\u003cspan class=\"n\"\u003ewiki\u003c/span\u003e\u003cspan class=\"o\"\u003e/\u003c/span\u003e\u003cspan class=\"n\"\u003eInstallation.\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 8\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003eThis\u003c/span\u003e \u003cspan class=\"n\"\u003ewarning\u003c/span\u003e \u003cspan class=\"n\"\u003emessage\u003c/span\u003e \u003cspan class=\"n\"\u003eshould\u003c/span\u003e \u003cspan class=\"n\"\u003enot\u003c/span\u003e \u003cspan class=\"n\"\u003eoccur\u003c/span\u003e \u003cspan class=\"n\"\u003eon\u003c/span\u003e \u003cspan class=\"n\"\u003eWindows\u003c/span\u003e \u003cspan class=\"n\"\u003eor\u003c/span\u003e \u003cspan class=\"n\"\u003eLinux.\u003c/span\u003e \u003cspan class=\"n\"\u003eIf\u003c/span\u003e \u003cspan class=\"n\"\u003eit\u003c/span\u003e \u003cspan class=\"n\"\u003edoes\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"n\"\u003eplease\u003c/span\u003e \u003cspan class=\"n\"\u003efile\u003c/span\u003e \u003cspan class=\"n\"\u003ea\u003c/span\u003e \u003cspan class=\"n\"\u003eGitHub\u003c/span\u003e \u003cspan class=\"n\"\u003eissue.\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 9\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"o\"\u003e**********\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e10\u003c/span\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e11\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nf\"\u003egetDTthreads\u003c/span\u003e\u003cspan class=\"p\"\u003e()\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e12\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e[1]\u003c/span\u003e \u003cspan class=\"m\"\u003e1\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch2 id=\"what-is-the-issue-here\"\u003eWhat is the issue here?\u003c/h2\u003e\n\u003cp\u003eOpenMP is an implementation of multithreading, and the Clang compiler that comes with Xcode on macOS lacks support for OpenMP. Apple has chosen not to include the \u003ccode\u003elibomp.dylib\u003c/code\u003e run-time library in their compiler. You can verify this by executing the following command.\u003c/p\u003e","title":"Using data.table with OpenMP support"},{"content":"My sequencing files are named according to the folowing pattern lane5651_AAGAGGCA_00h_Cell_WT3_L008_R1.fastq.gz. I would like to capture the Sample ID as 00h_Cell_WT3 in order to name all downstream files accordingly. To this end, I wrote the following snippet:\n1#!/usr/bin/env nextflow 2 3// fastq files are stored in reads as paired ends R1 and R4 4params.reads = \u0026#39;reads/lane*_*_*_*_R{1,4}.fastq.gz\u0026#39; 5 6Channel 7 .fromFilePairs(params.reads, flat: true) 8 .map { prefix, file1, file2 -\u0026gt; tuple(getSampleID(prefix), file1, file2) } 9 .set { samples_ch } 10 11def getSampleID( file ) { 12 // using RegEx to extract the SampleID 13 // in paried ends, fromFilePairs (with flat: true) returns a triplate 14 // where the first item is the filename without `R{1,4}.fastq.gz` 15 // thus the RegEx needs to be adjusted as follow 16 regexpPE = /([a-z]{4}[0-9]{4})_([A-Z]{8})_(.+)_(L[0-9]{3})/ 17 (file =~ regexpPE)[0][3] 18} 19 20process printNames { 21 input: 22 set sampleId, forward_reads, reverse_reads from samples_ch 23 24 output: 25 stdout result 26 27 \u0026#34;\u0026#34;\u0026#34; 28 echo $sampleId \u0026#39;and\u0026#39; $forward_reads \u0026#39;and\u0026#39; $reverse_reads 29 \u0026#34;\u0026#34;\u0026#34; 30} 31 32result.subscribe { println it } For single ends experiments, the following snippet can be used:\n1#!/usr/bin/env nextflow 2 3// fastq files are stored in reads as single ends R1 4params.reads = \u0026#39;reads/lane*_*_*_*_R1.fastq.gz\u0026#39; 5 6Channel 7 .fromPath(params.reads) 8 .map { sample -\u0026gt; tuple(getLibraryId(sample), sample) } 9 .set { samples_ch } 10 11def getLibraryId( file ) { 12 regexp = /([a-z]{4}[0-9]{4})_([A-Z]{8})_(.+)_(L[0-9]{3})_(R[1234])(.fastq.gz)/ 13 (file.name =~ regexp)[0][3] 14} 15 16process printNames { 17\tinput: 18\tset sampleId, file from samples_ch 19 20\toutput: 21\tstdout result 22 23\t\u0026#34;\u0026#34;\u0026#34; 24 echo $sampleId \u0026#39;and\u0026#39; $file 25\t\u0026#34;\u0026#34;\u0026#34; 26} 27 28result.subscribe { println it } ","permalink":"https://firas.phd/posts/regex_groovy/","summary":"\u003cp\u003eMy sequencing files are named according to the folowing pattern \u003ccode\u003elane5651_AAGAGGCA_00h_Cell_WT3_L008_R1.fastq.gz\u003c/code\u003e. I would like to capture the \u003ccode\u003eSample ID\u003c/code\u003e as \u003ccode\u003e00h_Cell_WT3\u003c/code\u003e in order to name all downstream files accordingly. To this end, I wrote the following snippet:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-java\" data-lang=\"java\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 1\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"err\"\u003e#\u003c/span\u003e\u003cspan class=\"o\"\u003e!/\u003c/span\u003e\u003cspan class=\"n\"\u003eusr\u003c/span\u003e\u003cspan class=\"o\"\u003e/\u003c/span\u003e\u003cspan class=\"n\"\u003ebin\u003c/span\u003e\u003cspan class=\"o\"\u003e/\u003c/span\u003e\u003cspan class=\"n\"\u003eenv\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003enextflow\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 2\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 3\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\u003c/span\u003e\u003cspan class=\"c1\"\u003e// fastq files are stored in reads as paired ends R1 and R4\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 4\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\u003c/span\u003e\u003cspan class=\"n\"\u003eparams\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003ereads\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"err\"\u003e\u0026#39;\u003c/span\u003e\u003cspan class=\"n\"\u003ereads\u003c/span\u003e\u003cspan class=\"o\"\u003e/\u003c/span\u003e\u003cspan class=\"n\"\u003elane\u003c/span\u003e\u003cspan class=\"o\"\u003e*\u003c/span\u003e\u003cspan class=\"n\"\u003e_\u003c/span\u003e\u003cspan class=\"o\"\u003e*\u003c/span\u003e\u003cspan class=\"n\"\u003e_\u003c/span\u003e\u003cspan class=\"o\"\u003e*\u003c/span\u003e\u003cspan class=\"n\"\u003e_\u003c/span\u003e\u003cspan class=\"o\"\u003e*\u003c/span\u003e\u003cspan class=\"n\"\u003e_R\u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"n\"\u003e1\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"n\"\u003e4\u003c/span\u003e\u003cspan class=\"p\"\u003e}.\u003c/span\u003e\u003cspan class=\"na\"\u003efastq\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003egz\u003c/span\u003e\u003cspan class=\"err\"\u003e\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 5\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 6\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\u003c/span\u003e\u003cspan class=\"n\"\u003eChannel\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 7\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e     \u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003efromFilePairs\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eparams\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003ereads\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eflat\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kc\"\u003etrue\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 8\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e     \u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003emap\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eprefix\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003efile1\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003efile2\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e-\u0026gt;\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003etuple\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003egetSampleID\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eprefix\u003c/span\u003e\u003cspan class=\"p\"\u003e),\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003efile1\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003efile2\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 9\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e     \u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003eset\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003esamples_ch\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e10\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e11\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\u003c/span\u003e\u003cspan class=\"n\"\u003edef\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nf\"\u003egetSampleID\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003efile\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e12\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"c1\"\u003e// using RegEx to extract the SampleID\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e13\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"c1\"\u003e// in paried ends, fromFilePairs (with flat: true) returns a triplate\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e14\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"c1\"\u003e//     where the first item is the filename without `R{1,4}.fastq.gz`\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e15\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"c1\"\u003e//     thus the RegEx needs to be adjusted as follow\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e16\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"n\"\u003eregexpPE\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e/\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"o\"\u003e[\u003c/span\u003e\u003cspan class=\"n\"\u003ea\u003c/span\u003e\u003cspan class=\"o\"\u003e-\u003c/span\u003e\u003cspan class=\"n\"\u003ez\u003c/span\u003e\u003cspan class=\"o\"\u003e]\u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"n\"\u003e4\u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"o\"\u003e[\u003c/span\u003e\u003cspan class=\"n\"\u003e0\u003c/span\u003e\u003cspan class=\"o\"\u003e-\u003c/span\u003e\u003cspan class=\"n\"\u003e9\u003c/span\u003e\u003cspan class=\"o\"\u003e]\u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"n\"\u003e4\u003c/span\u003e\u003cspan class=\"p\"\u003e})\u003c/span\u003e\u003cspan class=\"n\"\u003e_\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"o\"\u003e[\u003c/span\u003e\u003cspan class=\"n\"\u003eA\u003c/span\u003e\u003cspan class=\"o\"\u003e-\u003c/span\u003e\u003cspan class=\"n\"\u003eZ\u003c/span\u003e\u003cspan class=\"o\"\u003e]\u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"n\"\u003e8\u003c/span\u003e\u003cspan class=\"p\"\u003e})\u003c/span\u003e\u003cspan class=\"n\"\u003e_\u003c/span\u003e\u003cspan class=\"p\"\u003e(.\u003c/span\u003e\u003cspan class=\"o\"\u003e+\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"n\"\u003e_\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eL\u003c/span\u003e\u003cspan class=\"o\"\u003e[\u003c/span\u003e\u003cspan class=\"n\"\u003e0\u003c/span\u003e\u003cspan class=\"o\"\u003e-\u003c/span\u003e\u003cspan class=\"n\"\u003e9\u003c/span\u003e\u003cspan class=\"o\"\u003e]\u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"n\"\u003e3\u003c/span\u003e\u003cspan class=\"p\"\u003e})\u003c/span\u003e\u003cspan class=\"o\"\u003e/\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e17\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003efile\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e=~\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eregexpPE\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"o\"\u003e[\u003c/span\u003e\u003cspan class=\"n\"\u003e0\u003c/span\u003e\u003cspan class=\"o\"\u003e][\u003c/span\u003e\u003cspan class=\"n\"\u003e3\u003c/span\u003e\u003cspan class=\"o\"\u003e]\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e18\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e19\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e20\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\u003c/span\u003e\u003cspan class=\"n\"\u003eprocess\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eprintNames\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e21\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"n\"\u003einput\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e22\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"n\"\u003eset\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003esampleId\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eforward_reads\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003ereverse_reads\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003efrom\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003esamples_ch\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e23\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e24\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nl\"\u003eoutput\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e25\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"n\"\u003estdout\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eresult\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e26\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e27\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;\u0026#34;\u0026#34;\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e28\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s\"\u003e    echo $sampleId \u0026#39;and\u0026#39; $forward_reads \u0026#39;and\u0026#39; $reverse_reads\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e29\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s\"\u003e    \u0026#34;\u0026#34;\u0026#34;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e30\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e31\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e32\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\u003c/span\u003e\u003cspan class=\"n\"\u003eresult\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"na\"\u003esubscribe\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eprintln\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"n\"\u003eit\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eFor single ends experiments, the following snippet can be used:\u003c/p\u003e","title":"Using RegEx to capture file name in Groovy/NextFlow"},{"content":"Multiplexing\nIndex 1 (i7) is always read. Index 2 (i5) is read only in dual index setting. Single indexing Dual indexing Single end (SE) R1, R2 R1, R2, R3 Paired end (PE) R1, R2, R4 R1, R2, R3, R4 Multiplexing - dual index reads - paired end ","permalink":"https://firas.phd/posts/illumina_sbs/","summary":"\u003cp\u003eMultiplexing\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eIndex 1 (i7) is always read.\u003c/li\u003e\n\u003cli\u003eIndex 2 (i5) is read only in dual index setting.\u003c/li\u003e\n\u003c/ul\u003e\n\u003ctable\u003e\n  \u003cthead\u003e\n      \u003ctr\u003e\n          \u003cth style=\"text-align: center\"\u003e\u003c/th\u003e\n          \u003cth style=\"text-align: center\"\u003eSingle indexing\u003c/th\u003e\n          \u003cth style=\"text-align: center\"\u003eDual indexing\u003c/th\u003e\n      \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n      \u003ctr\u003e\n          \u003ctd style=\"text-align: center\"\u003eSingle end (SE)\u003c/td\u003e\n          \u003ctd style=\"text-align: center\"\u003eR1, R2\u003c/td\u003e\n          \u003ctd style=\"text-align: center\"\u003eR1, R2, R3\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd style=\"text-align: center\"\u003ePaired end (PE)\u003c/td\u003e\n          \u003ctd style=\"text-align: center\"\u003eR1, R2, R4\u003c/td\u003e\n          \u003ctd style=\"text-align: center\"\u003eR1, R2, R3, R4\u003c/td\u003e\n      \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"/images/Illumina_SBS/Illumina_SBS_1.png\"\u003e\u003c/p\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"/images/Illumina_SBS/Illumina_SBS_2.png\"\u003e\u003c/p\u003e\n\u003cp\u003eMultiplexing - dual index reads - paired end\n\u003cimg loading=\"lazy\" src=\"/images/Illumina_SBS/Illumina_SBS_3.png\"\u003e\u003c/p\u003e","title":"An overview of Illumina multiplexing"},{"content":"To prepare genome reference in FASTA format for mouse assembly NCBI37/mm9, we have two options:\nFrom UCSC Using the mm9 assembly from UCSC golden Path. Do not use the masked file chromFaMasked.tar.gz! 1# download `chromFa.tar.gz ` from UCSC golden path 2wget http://hgdownload.cse.ucsc.edu/goldenPath/mm9/bigZips/chromFa.tar.gz 3#or 4sync -avzP rsync://hgdownload.cse.ucsc.edu/goldenPath/mm9/bigZips/chromFa.tar.gz . 5 6# uncompress the downloaded file 7tar -xvzf chromFa.tar.gz 8 9# remove `*random.fa` chromosomes 10rm -rf *_random.fa 11 12# concatenate all FASTA files into a single file 13cat *.fa \u0026gt; mm9.fa 14 15# index the concatenated .fa file using `samtools` 16module load samtools # required at HPC 17samtools faidx mm9.fa From Ensembl Using Mus Musculus release-67 (Ensembl) 1# download the following folder from Ensembl 2lftp ftp://ftp.ensembl.org/pub/release-67/fasta/mus_musculus/dna/ 3mget * 4 5# uncompress the downloaded *.fa.gz files 6gzip -d *.fa.gz 7 8# delete the masked version of the genome sequence which contains \u0026#39;_rm\u0026#39; in the name 9rm -rf *_rm* 10 11# concatenate all FASTA files into a single file 12cat *.fa \u0026gt; mm9Ensembl.fa 13 14# index the concatenated .fa file using `samtools` 15module load samtools # required at HPC 16samtools faidx mm9Ensembl.fa One of the main differences between the two sources, which is very important for downstream applications, is chromosome annotation. Here is a comparison between the header lines, also known as the identifier or description lines, used in the FASTA files from both sources. While UCSC uses the chr prefix in front of the chromosome number, Ensembl merely uses the chromosome number. 1# mm9 from UCSC 2~/f/G/N/mm9_fasta ❯❯❯ cat mm9.fa | grep \u0026#39;\u0026gt;\u0026#39; 3\u0026gt;chr10 4\u0026gt;chr11 5\u0026gt;chr12 6\u0026gt;chr13 7\u0026gt;chr14 8\u0026gt;chr15 9\u0026gt;chr16 10\u0026gt;chr17 11\u0026gt;chr18 12\u0026gt;chr19 13\u0026gt;chr1 14\u0026gt;chr2 15\u0026gt;chr3 16\u0026gt;chr4 17\u0026gt;chr5 18\u0026gt;chr6 19\u0026gt;chr7 20\u0026gt;chr8 21\u0026gt;chr9 22\u0026gt;chrM 23\u0026gt;chrX 24\u0026gt;chrY 25 26# ENS67 from Ensembl 27~/f/G/N/ENS67_fasta ❯❯❯ cat ENS67.fa | grep \u0026#39;\u0026gt;\u0026#39; 28\u0026gt;10 dna:chromosome chromosome:NCBIM37:10:1:129993255:1 29\u0026gt;11 dna:chromosome chromosome:NCBIM37:11:1:121843856:1 30\u0026gt;12 dna:chromosome chromosome:NCBIM37:12:1:121257530:1 31\u0026gt;13 dna:chromosome chromosome:NCBIM37:13:1:120284312:1 32\u0026gt;14 dna:chromosome chromosome:NCBIM37:14:1:125194864:1 33\u0026gt;15 dna:chromosome chromosome:NCBIM37:15:1:103494974:1 34\u0026gt;16 dna:chromosome chromosome:NCBIM37:16:1:98319150:1 35\u0026gt;17 dna:chromosome chromosome:NCBIM37:17:1:95272651:1 36\u0026gt;18 dna:chromosome chromosome:NCBIM37:18:1:90772031:1 37\u0026gt;19 dna:chromosome chromosome:NCBIM37:19:1:61342430:1 38\u0026gt;1 dna:chromosome chromosome:NCBIM37:1:1:197195432:1 39\u0026gt;2 dna:chromosome chromosome:NCBIM37:2:1:181748087:1 40\u0026gt;3 dna:chromosome chromosome:NCBIM37:3:1:159599783:1 41\u0026gt;4 dna:chromosome chromosome:NCBIM37:4:1:155630120:1 42\u0026gt;5 dna:chromosome chromosome:NCBIM37:5:1:152537259:1 43\u0026gt;6 dna:chromosome chromosome:NCBIM37:6:1:149517037:1 44\u0026gt;7 dna:chromosome chromosome:NCBIM37:7:1:152524553:1 45\u0026gt;8 dna:chromosome chromosome:NCBIM37:8:1:131738871:1 46\u0026gt;9 dna:chromosome chromosome:NCBIM37:9:1:124076172:1 47\u0026gt;MT dna:chromosome chromosome:NCBIM37:MT:1:16299:1 48\u0026gt;X dna:chromosome chromosome:NCBIM37:X:1:166650296:1 49\u0026gt;Y dna:chromosome chromosome:NCBIM37:Y:1:15902555:1 ","permalink":"https://firas.phd/posts/genome_reference/","summary":"\u003cp\u003eTo prepare genome reference in FASTA format for mouse assembly NCBI37/mm9, we have two options:\u003c/p\u003e\n\u003ch3 id=\"from-ucsc\"\u003eFrom UCSC\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003eUsing the mm9 assembly from \u003ca href=\"http://hgdownload.cse.ucsc.edu/goldenPath/mm9/bigZips/\"\u003eUCSC golden Path\u003c/a\u003e.\u003c/li\u003e\n\u003cli\u003eDo not use the masked file \u003ccode\u003echromFaMasked.tar.gz\u003c/code\u003e!\u003c/li\u003e\n\u003c/ul\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 1\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# download `chromFa.tar.gz ` from UCSC golden path\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 2\u003c/span\u003e\u003cspan class=\"cl\"\u003ewget http://hgdownload.cse.ucsc.edu/goldenPath/mm9/bigZips/chromFa.tar.gz\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 3\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e#or\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 4\u003c/span\u003e\u003cspan class=\"cl\"\u003esync -avzP rsync://hgdownload.cse.ucsc.edu/goldenPath/mm9/bigZips/chromFa.tar.gz .\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 5\u003c/span\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 6\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# uncompress the downloaded file\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 7\u003c/span\u003e\u003cspan class=\"cl\"\u003etar -xvzf chromFa.tar.gz\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 8\u003c/span\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 9\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# remove `*random.fa` chromosomes\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e10\u003c/span\u003e\u003cspan class=\"cl\"\u003erm -rf *_random.fa\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e11\u003c/span\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e12\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# concatenate all FASTA files into a single file\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e13\u003c/span\u003e\u003cspan class=\"cl\"\u003ecat *.fa \u0026gt; mm9.fa\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e14\u003c/span\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e15\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# index the concatenated .fa file using `samtools`\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e16\u003c/span\u003e\u003cspan class=\"cl\"\u003emodule load samtools \u003cspan class=\"c1\"\u003e# required at HPC\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e17\u003c/span\u003e\u003cspan class=\"cl\"\u003esamtools faidx mm9.fa\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch3 id=\"from-ensembl\"\u003eFrom Ensembl\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003eUsing Mus Musculus release-67 (\u003ca href=\"#ZgotmplZ\"\u003eEnsembl\u003c/a\u003e)\u003c/li\u003e\n\u003c/ul\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 1\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# download the following folder from Ensembl\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 2\u003c/span\u003e\u003cspan class=\"cl\"\u003elftp ftp://ftp.ensembl.org/pub/release-67/fasta/mus_musculus/dna/\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 3\u003c/span\u003e\u003cspan class=\"cl\"\u003emget *\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 4\u003c/span\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 5\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# uncompress the downloaded *.fa.gz files\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 6\u003c/span\u003e\u003cspan class=\"cl\"\u003egzip -d *.fa.gz\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 7\u003c/span\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 8\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# delete the masked version of the genome sequence which contains \u0026#39;_rm\u0026#39; in the name\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 9\u003c/span\u003e\u003cspan class=\"cl\"\u003erm -rf *_rm*\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e10\u003c/span\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e11\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# concatenate all FASTA files into a single file\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e12\u003c/span\u003e\u003cspan class=\"cl\"\u003ecat *.fa \u0026gt; mm9Ensembl.fa\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e13\u003c/span\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e14\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# index the concatenated .fa file using `samtools`\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e15\u003c/span\u003e\u003cspan class=\"cl\"\u003emodule load samtools \u003cspan class=\"c1\"\u003e# required at HPC\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e16\u003c/span\u003e\u003cspan class=\"cl\"\u003esamtools faidx mm9Ensembl.fa\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cul\u003e\n\u003cli\u003eOne of the main differences between the two sources, which is very important for downstream applications, is chromosome annotation.\u003c/li\u003e\n\u003cli\u003eHere is a comparison between the header lines, also known as the identifier or description lines, used in the FASTA files from both sources.\u003c/li\u003e\n\u003cli\u003eWhile UCSC uses the \u003ccode\u003echr\u003c/code\u003e prefix in front of the chromosome number, Ensembl merely uses the chromosome number.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-shell\" data-lang=\"shell\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 1\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# mm9 from UCSC\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 2\u003c/span\u003e\u003cspan class=\"cl\"\u003e~/f/G/N/mm9_fasta ❯❯❯ cat mm9.fa \u003cspan class=\"p\"\u003e|\u003c/span\u003e grep \u003cspan class=\"s1\"\u003e\u0026#39;\u0026gt;\u0026#39;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 3\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u0026gt;chr10\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 4\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u0026gt;chr11\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 5\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u0026gt;chr12\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 6\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u0026gt;chr13\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 7\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u0026gt;chr14\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 8\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u0026gt;chr15\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 9\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u0026gt;chr16\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e10\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u0026gt;chr17\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e11\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u0026gt;chr18\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e12\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u0026gt;chr19\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e13\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u0026gt;chr1\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e14\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u0026gt;chr2\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e15\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u0026gt;chr3\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e16\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u0026gt;chr4\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e17\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u0026gt;chr5\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e18\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u0026gt;chr6\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e19\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u0026gt;chr7\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e20\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u0026gt;chr8\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e21\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u0026gt;chr9\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e22\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u0026gt;chrM\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e23\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u0026gt;chrX\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e24\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u0026gt;chrY\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e25\u003c/span\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e26\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# ENS67 from Ensembl\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e27\u003c/span\u003e\u003cspan class=\"cl\"\u003e~/f/G/N/ENS67_fasta ❯❯❯ cat ENS67.fa \u003cspan class=\"p\"\u003e|\u003c/span\u003e grep \u003cspan class=\"s1\"\u003e\u0026#39;\u0026gt;\u0026#39;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e28\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u0026gt;10 dna:chromosome chromosome:NCBIM37:10:1:129993255:1\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e29\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u0026gt;11 dna:chromosome chromosome:NCBIM37:11:1:121843856:1\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e30\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u0026gt;12 dna:chromosome chromosome:NCBIM37:12:1:121257530:1\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e31\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u0026gt;13 dna:chromosome chromosome:NCBIM37:13:1:120284312:1\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e32\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u0026gt;14 dna:chromosome chromosome:NCBIM37:14:1:125194864:1\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e33\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u0026gt;15 dna:chromosome chromosome:NCBIM37:15:1:103494974:1\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e34\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u0026gt;16 dna:chromosome chromosome:NCBIM37:16:1:98319150:1\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e35\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u0026gt;17 dna:chromosome chromosome:NCBIM37:17:1:95272651:1\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e36\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u0026gt;18 dna:chromosome chromosome:NCBIM37:18:1:90772031:1\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e37\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u0026gt;19 dna:chromosome chromosome:NCBIM37:19:1:61342430:1\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e38\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u0026gt;1 dna:chromosome chromosome:NCBIM37:1:1:197195432:1\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e39\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u0026gt;2 dna:chromosome chromosome:NCBIM37:2:1:181748087:1\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e40\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u0026gt;3 dna:chromosome chromosome:NCBIM37:3:1:159599783:1\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e41\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u0026gt;4 dna:chromosome chromosome:NCBIM37:4:1:155630120:1\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e42\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u0026gt;5 dna:chromosome chromosome:NCBIM37:5:1:152537259:1\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e43\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u0026gt;6 dna:chromosome chromosome:NCBIM37:6:1:149517037:1\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e44\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u0026gt;7 dna:chromosome chromosome:NCBIM37:7:1:152524553:1\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e45\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u0026gt;8 dna:chromosome chromosome:NCBIM37:8:1:131738871:1\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e46\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u0026gt;9 dna:chromosome chromosome:NCBIM37:9:1:124076172:1\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e47\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u0026gt;MT dna:chromosome chromosome:NCBIM37:MT:1:16299:1\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e48\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u0026gt;X dna:chromosome chromosome:NCBIM37:X:1:166650296:1\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e49\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u0026gt;Y dna:chromosome chromosome:NCBIM37:Y:1:15902555:1\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e","title":"Preparing genome reference in FASTA format"},{"content":"Here, I summarise how I setup my macOS for the purpose of Data Science. Ultimately, I should invest some time into automating the process.\nInstall Command Line Developer Tools 1xcode-select --install Install HomeBrew 1/usr/bin/ruby -e \u0026#34;$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)\u0026#34; Turn off analytic (optional) 1brew analytics off Install iTerm 1brew install iterm Configure iTerm To download different colour schemes, visit iTerm Themes Install Go2Sehll Download and install from Go2Shell, then configure it to work with iTerm:\nInstall git 1brew install git Configure git 1git config --global user.name \u0026#34;userName\u0026#34; 2git config --global user.email \u0026#34;email@mailprovider.com\u0026#34; 3git config --global core.excludesfile \u0026#39;~/.gitignore\u0026#39; Configure ssh key 1mkdir ~/.ssh #if it does not already exist 2chmod 700 ~/.ssh 3cd ~/.ssh 4ssh-keygen -t rsa -C \u0026#34;email@mailprovider.com\u0026#34; 5enter a keyname 6enter a passphrase 7cat ~/.ssh/keyname.pub Copy the key to GitHub. Go to GitHub Settings. From the left margin, choose SSH and GPG keys and click on New SSH key. Paste the value copied above and give the key a name.\n1eval `ssh-agent -s` # if needed 2ssh-add ~/.ssh/KEYNAME Install zsh 1brew install zsh 2echo \u0026#34;$(which zsh)\u0026#34; 3sudo vi /etc/shells Edit /etc/shells/ and add the path /usr/local/bin/zsh at the end of the file.\n1chsh -s $(which zsh) Install prezto 1git clone --recursive https://github.com/sorin-ionescu/prezto.git \u0026#34;${ZDOTDIR:-$HOME}/.zprezto\u0026#34; Configure prezto Create soft links ln -s to all the files found in ~/.zprezto/runcoms/ and change each link\u0026rsquo;s NAME to .NAME:\n1cd ~ 2ln -s .zprezto/runcoms/z* . 3mv zlogin .zlogin 4mv zlogout .zlogout 5mv zpreztorc .zpreztorc 6mv zprofile .zprofile 7mv zshenv .zshenv 8mv zshrc .zshrc Prezto has many useful modules and here is a list of all modules available. You can edit .zpreztorc to enable any of them:\n1# Set the Prezto modules to load (browse modules). 2# The order matters. 3zstyle \u0026#39;:prezto:load\u0026#39; pmodule \\ 4 \u0026#39;environment\u0026#39; \\ 5 \u0026#39;terminal\u0026#39; \\ 6 \u0026#39;editor\u0026#39; \\ 7 \u0026#39;history\u0026#39; \\ 8 \u0026#39;directory\u0026#39; \\ 9 \u0026#39;spectrum\u0026#39; \\ 10 \u0026#39;utility\u0026#39; \\ 11 \u0026#39;completion\u0026#39; \\ 12 \u0026#39;git\u0026#39; \\ 13 \u0026#39;archive\u0026#39; \\ 14 \u0026#39;osx\u0026#39; \\ 15 \u0026#39;python\u0026#39; \\ 16 \u0026#39;ruby\u0026#39; \\ 17 \u0026#39;fasd\u0026#39; \\ 18 \u0026#39;autosuggestions\u0026#39; \\ 19 \u0026#39;command-not-found\u0026#39; \\ 20 \u0026#39;dnf\u0026#39; \\ 21 \u0026#39;dpkg\u0026#39; \\ 22 \u0026#39;emacs\u0026#39; \\ 23 \u0026#39;gnu-utility\u0026#39; \\ 24 \u0026#39;gpg\u0026#39; \\ 25 \u0026#39;haskell\u0026#39; \\ 26 \u0026#39;helper\u0026#39; \\ 27 \u0026#39;homebrew\u0026#39; \\ 28 \u0026#39;macports\u0026#39; \\ 29 \u0026#39;node\u0026#39; \\ 30 \u0026#39;ocaml\u0026#39; \\ 31 \u0026#39;pacman\u0026#39; \\ 32 \u0026#39;perl\u0026#39; \\ 33 \u0026#39;rails\u0026#39; \\ 34 \u0026#39;rsync\u0026#39; \\ 35 \u0026#39;screen\u0026#39; \\ 36 \u0026#39;ssh\u0026#39; \\ 37 \u0026#39;tmux\u0026#39; \\ 38 \u0026#39;wakeonlan\u0026#39; \\ 39 \u0026#39;yum\u0026#39; \\ 40 \u0026#39;syntax-highlighting\u0026#39; \\ 41 \u0026#39;history-substring-search\u0026#39; \\ 42 \u0026#39;prompt\u0026#39; Install Sublime Text 1brew cask install sublime-text Configure Sublime Text 1sudo ln -s /Applications/Sublime\\ Text.app/Contents/SharedSupport/bin/subl /usr/local/bin/subl And then install Package Control to enable various useful packages.\nNow, using the keyboard shortcut Cmd + Shift + p to invoke the Command Palette.\nType Install Package and hit enter, then type the name of the package to be installed. Visit Package Control website to search for useful packages.\nInstall R 1brew tap homebrew/science 2brew install openblas 3brew install r --with-openblas Install RStudio (Preview version) 1brew install Caskroom/cask/rstudio-preview Hints Open finder from terminal 1open . Show/hide hidden files 1defaults write com.apple.finder AppleShowAllFiles YES # to show hidden files 2defaults write com.apple.finder AppleShowAllFiles NO # to hide hidden files Then, press alt while clicking on Finder icon to Relunch Finder.\n","permalink":"https://firas.phd/posts/macos_setup/","summary":"\u003cp\u003eHere, I summarise how I setup my macOS for the purpose of Data Science. Ultimately, I should invest some time into automating the process.\u003c/p\u003e\n\u003ch3 id=\"install-command-line-developer-tools\"\u003eInstall Command Line Developer Tools\u003c/h3\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e1\u003c/span\u003e\u003cspan class=\"cl\"\u003excode-select --install\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch3 id=\"install-homebrew\"\u003eInstall HomeBrew\u003c/h3\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e1\u003c/span\u003e\u003cspan class=\"cl\"\u003e/usr/bin/ruby -e \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"k\"\u003e$(\u003c/span\u003ecurl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install\u003cspan class=\"k\"\u003e)\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch3 id=\"turn-off-analytic-optional\"\u003eTurn off analytic (optional)\u003c/h3\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e1\u003c/span\u003e\u003cspan class=\"cl\"\u003ebrew analytics off\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch3 id=\"install-iterm\"\u003eInstall iTerm\u003c/h3\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e1\u003c/span\u003e\u003cspan class=\"cl\"\u003ebrew install iterm\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch3 id=\"configure-iterm\"\u003eConfigure iTerm\u003c/h3\u003e\n\u003cp\u003e\u003cimg alt=\"iTerm Configure1\" loading=\"lazy\" src=\"/images/iTermConfig1.png\"\u003e\n\u003cimg alt=\"iTerm Configure2\" loading=\"lazy\" src=\"/images/iTermConfig2.png\"\u003e\nTo download different colour schemes, visit \u003ca href=\"http://iterm2colorschemes.com/\"\u003eiTerm Themes\u003c/a\u003e\n\u003cimg alt=\"iTerm Configure3\" loading=\"lazy\" src=\"/images/iTermConfig3.png\"\u003e\u003c/p\u003e\n\u003ch3 id=\"install-go2sehll\"\u003eInstall Go2Sehll\u003c/h3\u003e\n\u003cp\u003eDownload and install from \u003ca href=\"http://zipzapmac.com/go2shell\"\u003eGo2Shell\u003c/a\u003e, then configure it to work with iTerm:\u003c/p\u003e","title":"macOS setup for data science"},{"content":"This posts shows how I setup a new Ubuntu for Data Science.\nUpdate Ubuntu 1sudo apt update 2sudo apt upgrade Configure apt It\u0026rsquo;s always a good idea to check the baseline setup. So, let\u0026rsquo;s check the current apt keys.\n1sudo apt-key list Then, let\u0026rsquo;s check the sources.\n1cat /etc/apt/sources.list 2cd /etc/apt/sources.list.d/ And finally, let\u0026rsquo;s check the users and group.\n1cut -d: -f1 /etc/passwd # will list all local users 2cat /etc/passwd # will list all local users with groups and other properties 3getent passwd # same as above 4 5cut -d: -f1 /etc/group # will list all local groups 6cat /etc/group # will list all local groups 7id -Gn 8groups 9 10getent group groupname # will list all members of groupaname 11id username # will list all the groups a particular username belongs to Configure Unity To get minimise window on clicking its icon:\n1gsettings set org.compiz.unityshell:/org/compiz/profiles/unity/plugins/unityshell/ launcher-minimize-window true Change folder view as tree\n1Files Preferences \u0026gt; Display \u0026gt; Navigate folders in a tree Install git 1sudo apt install git Configure git 1git config --global user.name \u0026#34;userName\u0026#34; 2git config --global user.email \u0026#34;email@mailprovider.com\u0026#34; 3git config --global core.excludesfile \u0026#39;~/.gitignore\u0026#39; Connect to GitHub using a SSH key 1mkdir ~/.ssh #if it does not already exist 2chmod 700 ~/.ssh 3cd ~/.ssh 4ssh-keygen -t rsa -C \u0026#34;email@mailprovider.com\u0026#34; 5enter a keyname 6enter a passphrase 7cat ~/.ssh/keyname.pub Copy the key to GitHub. Go to GitHub Settings. From the left margin, choose SSH and GPG keys and click on New SSH key. Paste the value copied above and give the key a name.\n1eval `ssh-agent -s` # if needed 2ssh-add ~/.ssh/KEYNAME Connect to external servers using SSH keys 1mkdir ~/.ssh #if it does not already exist 2chmod 700 ~/.ssh 3cd ~/.ssh 4ssh-keygen -t rsa 5enter a keyname 6enter a passphrase 7ssh-copy-id -i [path to rsa file] USER@SERVER Install SSHFS SSHFS allows you to mount a remote filesystem using SFTP.\n1sudo apt install sshfs Create a mounting point\n1sudo sshfs USER@SERVER:/ /mnt/foo/ -o allow_other,IdentityFile=/home/USER/.ssh/SERVER_rsa Disconnect a mounting point\n1sudo umount /mnt/foo or to mount the drivers without sudo everytime using an admin account\n1sudo chown firas: /mnt/foo/ Then using your account\n1chmod 700 foo Now, you can do\n1sshfs USER@SERVER:/ /mnt/foo -o IdentityFile=/home/USER/.ssh/SERVER_rsa Keeping allow_other will cause the error: fusermount: option allow_other only allowed if 'user_allow_other' is set in /etc/fuse.conf\nInstall CIFS 1sudo apt-get install cifs-utils To mount windows share (SMB) as local disk:\n1id 2sudo mount //WINSRV/SHARE -t cifs -o uid=1000,gid=1000,username=USER,domain=DOMAIN /mnt/foo 3sudo mount -t cifs //WINSRV/SHARE ~/localmount -o user=userid,pass=mypass,cache=loose,noperm,dir_mode=0777,file_mode=0777 Install Remmina 1sudo apt-add-repository ppa:remmina-ppa-team/remmina-next 2sudo apt update 3sudo apt install remmina remmina-plugin-rdp libfreerdp-plugins-standard Configure Terminal 1 /* 8 normal colors */ 2 [0] = \u0026#34;#2f1e2e\u0026#34;, /* black */ 3 [1] = \u0026#34;#ef6155\u0026#34;, /* red */ 4 [2] = \u0026#34;#48b685\u0026#34;, /* green */ 5 [3] = \u0026#34;#fec418\u0026#34;, /* yellow */ 6 [4] = \u0026#34;#06b6ef\u0026#34;, /* blue */ 7 [5] = \u0026#34;#815ba4\u0026#34;, /* magenta */ 8 [6] = \u0026#34;#5bc4bf\u0026#34;, /* cyan */ 9 [7] = \u0026#34;#a39e9b\u0026#34;, /* white */ 10 11 /* 8 bright colors */ 12 [8] = \u0026#34;#776e71\u0026#34;, /* black */ 13 [9] = \u0026#34;#ef6155\u0026#34;, /* red */ 14 [10] = \u0026#34;#48b685\u0026#34;, /* green */ 15 [11] = \u0026#34;#fec418\u0026#34;, /* yellow */ 16 [12] = \u0026#34;#06b6ef\u0026#34;, /* blue */ 17 [13] = \u0026#34;#815ba4\u0026#34;, /* magenta */ 18 [14] = \u0026#34;#5bc4bf\u0026#34;, /* cyan */ 19 [15] = \u0026#34;#e7e9db\u0026#34;, /* white */ 20 21 /* special colors */ 22 [256] = \u0026#34;#2f1e2e\u0026#34;, /* background */ 23 [257] = \u0026#34;#a39e9b\u0026#34;, /* foreground */ Install zsh and prezto 1sudo apt update 2sudo apt install zsh 3chsh -s $(which zsh) # to change to zsh, don\u0026#39;t use sudo otherwise it will change it for root 4sudo vi /etc/passwd # just to check that now :/home/USERNAME:/usr/bin/zsh 5# logout and login 6git clone --recursive https://github.com/sorin-ionescu/prezto.git \u0026#34;${ZDOTDIR:-$HOME}/.zprezto\u0026#34; Configure prezto 1cd ~ 2ln -s .zprezto/runcoms/z* . 3mv zlogin .zlogin 4mv zlogout .zlogout 5mv zpreztorc .zpreztorc 6mv zprofile .zprofile 7mv zshenv .zshenv 8mv zshrc .zshrc Prezto has many useful modules and here is a list of all modules available. You can edit .zpreztorc to enable any of them:\n1# Set the Prezto modules to load (browse modules). 2# The order matters. 3zstyle \u0026#39;:prezto:load\u0026#39; pmodule \\ 4 \u0026#39;environment\u0026#39; \\ 5 \u0026#39;terminal\u0026#39; \\ 6 \u0026#39;editor\u0026#39; \\ 7 \u0026#39;history\u0026#39; \\ 8 \u0026#39;directory\u0026#39; \\ 9 \u0026#39;spectrum\u0026#39; \\ 10 \u0026#39;utility\u0026#39; \\ 11 \u0026#39;completion\u0026#39; \\ 12 \u0026#39;git\u0026#39; \\ 13 \u0026#39;archive\u0026#39; \\ 14 \u0026#39;osx\u0026#39; \\ 15 \u0026#39;python\u0026#39; \\ 16 \u0026#39;ruby\u0026#39; \\ 17 \u0026#39;fasd\u0026#39; \\ 18 \u0026#39;autosuggestions\u0026#39; \\ 19 \u0026#39;command-not-found\u0026#39; \\ 20 \u0026#39;dnf\u0026#39; \\ 21 \u0026#39;dpkg\u0026#39; \\ 22 \u0026#39;emacs\u0026#39; \\ 23 \u0026#39;gnu-utility\u0026#39; \\ 24 \u0026#39;gpg\u0026#39; \\ 25 \u0026#39;haskell\u0026#39; \\ 26 \u0026#39;helper\u0026#39; \\ 27 \u0026#39;homebrew\u0026#39; \\ 28 \u0026#39;macports\u0026#39; \\ 29 \u0026#39;node\u0026#39; \\ 30 \u0026#39;ocaml\u0026#39; \\ 31 \u0026#39;pacman\u0026#39; \\ 32 \u0026#39;perl\u0026#39; \\ 33 \u0026#39;rails\u0026#39; \\ 34 \u0026#39;rsync\u0026#39; \\ 35 \u0026#39;screen\u0026#39; \\ 36 \u0026#39;ssh\u0026#39; \\ 37 \u0026#39;tmux\u0026#39; \\ 38 \u0026#39;wakeonlan\u0026#39; \\ 39 \u0026#39;yum\u0026#39; \\ 40 \u0026#39;syntax-highlighting\u0026#39; \\ 41 \u0026#39;history-substring-search\u0026#39; \\ 42 \u0026#39;prompt\u0026#39; Install R and RStudio To get the most updated version of R, add CRAN repository 1sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys E298A3A825C0D65DFD57CBB651716619E084DAB9 2sudo add-apt-repository \u0026#39;deb [arch=amd64,i386] https://cran.rstudio.com/bin/linux/ubuntu xenial/\u0026#39; 3sudo apt update 4sudo apt install r-base 5sudo apt install r-base-dev To Install Rstudio,\n1sudo dpkg -i libgstreamer0.10-0_0.10.36-1.5_amd64.deb 2sudo dpkg -i libgstreamer-plugins-base0.10-0_0.10.36-2_amd64.deb 3sudo apt install libjpeg62 libedit2 4sudo dpkg -i rstudio-1.1.383-amd64.deb Install Python virutal environment Ubuntu 16.04 comes with Python and Python3 installed.\nInstall and update pip3:\n1sudo apt install python3-pip 2pip3 install --upgrade pip setuptools wheel Install virtualenv and virtualenvwrapper\n1sudo apt install virtualenv virtualenvwrapper Add the following lines to .zprofile:\n1# needed for virtualenvwrapper 2export WORKON_HOME=$HOME/.virtualenvs 3export PROJECT_HOME=$HOME/Projects 4export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3 5export VIRTUALENVWRAPPER_VIRTUALENV=/usr/bin/virtualenv 6export PIP_REQUIRE_VIRTUALENV=true 7source /usr/local/bin/virtualenvwrapper.sh Install Jupyter notebook 1mkproject -p `which python3` Jupyter 2pip3 install jupyter Install Sublime Text 3 Follow the steps mentioned in the Sublime 3 Docs\nInstall the GPG key:\n1wget -qO - https://download.sublimetext.com/sublimehq-pub.gpg | sudo apt-key add - Ensure apt is set up to work with https sources:\n1sudo apt-get install apt-transport-https Select the channel to use:\n1echo \u0026#34;deb https://download.sublimetext.com/ apt/stable/\u0026#34; | sudo tee /etc/apt/sources.list.d/sublime-text.list Update apt sources and install Sublime Text:\n1sudo apt-get update 2sudo apt-get install sublime-text Tips:\nYou can start Sublime Text from the terminal using subl command.\nIf Sublime Text is not showing in the \u0026lsquo;open with\u0026rsquo; menu, then follow these steps to fix it:\nCopy the contents of /usr/share/applications/sublime_text.desktop to ~/.local/share/applications/sublime_text.desktop Then in the terminal type: sudo update-desktop-database Install Source Code Pro Download the latest release from GitHub repo\n1 2# For system wide installation, copy the fonts to `/usr/share/fonts` and run `sudo fc-cache` to rebuild the font cache, 3# For a local installation: create `~/.fonts` if not already exists, copy the font files here, then rebuild the font cache. 4 5mkdir ~/.fonts 6cd ~/.fonts 7# copy all OTF files to this directory 8cp ~/Download/*.otf ~/.fonts 9fc-cache -f -v Install vim 1sudo apt install vim Install SnapGene viewer 1wget http://www.snapgene.com//products/snapgene_viewer/download.php?\u0026amp;majorRelease=4.0\u0026amp;minorRelease=4.0.6\u0026amp;os=linux_deb 2sudo dpkg -i snapgene_viewer_4.0.6_linux.deb 3sudo apt-get install ttf-mscorefonts-installer 4fc-cache -f -v # to rebuild font cache manually Install htop 1sudo apt install htop Install Zotero Download Zotero from here, and unzip the downloaded file\n1tar -xvjf Zotero-5.0.21_linux-x86_64.tar.bz2 where -j is equal to --bzip2. Rename the folder and move it to /opt/.\nTo create a shortcut that you can use in Unity launcher, do the following:\n1cd /usr/share/applications 2sudo ln -s /opt/Zotero/zotero.desktop . In Ubuntu 18.04 and gnome desktop:\n1cd ~/Downloads 2wget https://download.zotero.org/client/release/5.0.47/Zotero-5.0.47_linux-x86_64.tar.bz2 3tar -xjvf Zotero-5.0.47_linux-x86_64.tar.bz2 4sudo mv Zotero_linux-x86_64/ /opt/zotero 5ln -s /opt/zotero/zotero.desktop ~/.local/share/applications/ Instal WPS Office 1wget http://ftp.debian.org/debian/pool/main/libp/libpng/libpng12-0_1.2.50-2+deb8u3_amd64.deb 2sudo dpkg -i libpng12-0_1.2.50-2+deb8u3_amd64.deb 3wget http://kdl1.cache.wps.com/ksodl/download/linux/a21//wps-office_10.1.0.5707-a21_amd64.deb 4sudo dpkg -i wps-office_10.1.0.5707-a21_amd64.deb Symbol fonts\n1cd /tmp 2git clone https://github.com/iamdh4/ttf-wps-fonts.git 3sudo mkdir /usr/share/fonts/wps-fonts 4sudo mv ttf-wps-fonts/* /usr/share/fonts/wps-fonts 5sudo chmod 644 /usr/share/fonts/wps-fonts/* 6sudo fc-cache -vfs 7rm -rf /tmp/ttf-wps-fonts Install Hugo To get the latest version of Hugo, use snap to install it.\n1sudo snap install hugo Install SNX (optional) 1sudo apt install libstdc++5:i386 libpam0g:i386 libx11-6:i386 2sh +x checkpoint vpn snx linux 800007075 Tips To open the directory from terminal 1nautilus . \u0026amp; 2xdg-open . \u0026amp; To reset Ubuntu Desktop to its default settings 1dconf reset -f / To enable En-GB spell checking in LibreOffice 1sudo apt install hunspell-en-gb To install a better screen shot manager Download the one that comes with Deepin distribution from here\n1sudo apt install python-xlib 2sudo dpkg -i deepin-scrot_2.0-0deepin_all.deb To preview a log file live 1tail -f ATACseq_TimePiont_diffbind_analysis.R.log Where -f is follow: output appended data as the file grows.\nInstall a graphical system load indicator for CPU, ram, etc.. 1sudo apt install indicator-multiload Install sushi-gnome\n1sudo apt-get install gnome-sushi Install Chromioum\n1sudo apt-get install chromium-browser ","permalink":"https://firas.phd/posts/ubuntu_setup/","summary":"\u003cp\u003eThis posts shows how I setup a new Ubuntu for Data Science.\u003c/p\u003e\n\u003ch3 id=\"update-ubuntu\"\u003eUpdate Ubuntu\u003c/h3\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e1\u003c/span\u003e\u003cspan class=\"cl\"\u003esudo apt update\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e2\u003c/span\u003e\u003cspan class=\"cl\"\u003esudo apt upgrade\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch3 id=\"configure-apt\"\u003eConfigure apt\u003c/h3\u003e\n\u003cp\u003eIt\u0026rsquo;s always a good idea to check the baseline setup. So, let\u0026rsquo;s check the current apt keys.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e1\u003c/span\u003e\u003cspan class=\"cl\"\u003esudo apt-key list\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThen, let\u0026rsquo;s check the sources.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e1\u003c/span\u003e\u003cspan class=\"cl\"\u003ecat /etc/apt/sources.list\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e2\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003ecd\u003c/span\u003e /etc/apt/sources.list.d/\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eAnd finally, let\u0026rsquo;s check the users and group.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 1\u003c/span\u003e\u003cspan class=\"cl\"\u003ecut -d: -f1 /etc/passwd \u003cspan class=\"c1\"\u003e# will list all local users\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 2\u003c/span\u003e\u003cspan class=\"cl\"\u003ecat /etc/passwd \u003cspan class=\"c1\"\u003e# will list all local users with groups and other properties\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 3\u003c/span\u003e\u003cspan class=\"cl\"\u003egetent passwd \u003cspan class=\"c1\"\u003e# same as above\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 4\u003c/span\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 5\u003c/span\u003e\u003cspan class=\"cl\"\u003ecut -d: -f1 /etc/group \u003cspan class=\"c1\"\u003e# will list all local groups\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 6\u003c/span\u003e\u003cspan class=\"cl\"\u003ecat /etc/group \u003cspan class=\"c1\"\u003e# will list all local groups\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 7\u003c/span\u003e\u003cspan class=\"cl\"\u003eid -Gn\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 8\u003c/span\u003e\u003cspan class=\"cl\"\u003egroups\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 9\u003c/span\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e10\u003c/span\u003e\u003cspan class=\"cl\"\u003egetent group groupname \u003cspan class=\"c1\"\u003e# will list all members of groupaname\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e11\u003c/span\u003e\u003cspan class=\"cl\"\u003eid username \u003cspan class=\"c1\"\u003e# will list all the groups a particular username belongs to\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch3 id=\"configure-unity\"\u003eConfigure Unity\u003c/h3\u003e\n\u003cp\u003eTo get minimise window on clicking its icon:\u003c/p\u003e","title":"Ubuntu setup for data science"},{"content":"To mount a network drive as a local one, you can use sshfs.\nGenerate SSH key 1$ mkdir ~/.ssh #if it does not already exist 2$ chmod 700 ~/.ssh 3$ cd ~/.ssh 4$ ssh-keygen -t rsa 5$ enter a keyname 6$ enter a passphrase 7$ ssh-copy-id -i [path to rsa file] USER@SERVER Where -i indicates where the rsa file is located. The .pub key is the one needed to be copied.\n[Optional] Start a SSH agent, and add the key to the SSH agent:\n1$ eval `ssh-agent -s` # if needed 2$ ssh-add ~/.ssh/KEY.pub Install sshfs For Ubuntu:\n1sudo apt install sshfs For Mac: Download both FUSE and SSHFS from here\nCreate a mounting point 1# Ubuntu 2sudo mkdir /mnt/foo 3 4# Mac 5sudo mkdir /Volumes/foo Connect to the remove server 1# Ubuntu 2sudo sshfs USER@SERVER:/ /mnt/foo/ -o allow_other,IdentityFile=/home/USER/.ssh/SERVER_rsa 3 4# Mac 5sudo sshfs USER@SERVER:/ /Volumes/foo/ -o allow_other,IdentityFile=/home/USER/.ssh/SERVER_rsa,defer_permissions Disconnect from the remove server 1# Ubuntu 2sudo umount /mnt/foo 3 4# Mac 5sudo umount /Volumes/foo ","permalink":"https://firas.phd/posts/macos_sshfs/","summary":"\u003cp\u003eTo mount a network drive as a local one, you can use sshfs.\u003c/p\u003e\n\u003ch3 id=\"generate-ssh-key\"\u003eGenerate SSH key\u003c/h3\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e1\u003c/span\u003e\u003cspan class=\"cl\"\u003e$ mkdir ~/.ssh \u003cspan class=\"c1\"\u003e#if it does not already exist\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e2\u003c/span\u003e\u003cspan class=\"cl\"\u003e$ chmod \u003cspan class=\"m\"\u003e700\u003c/span\u003e ~/.ssh\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e3\u003c/span\u003e\u003cspan class=\"cl\"\u003e$ \u003cspan class=\"nb\"\u003ecd\u003c/span\u003e ~/.ssh\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e4\u003c/span\u003e\u003cspan class=\"cl\"\u003e$ ssh-keygen -t rsa\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e5\u003c/span\u003e\u003cspan class=\"cl\"\u003e$ enter a keyname\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e6\u003c/span\u003e\u003cspan class=\"cl\"\u003e$ enter a passphrase\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e7\u003c/span\u003e\u003cspan class=\"cl\"\u003e$ ssh-copy-id -i \u003cspan class=\"o\"\u003e[\u003c/span\u003epath to rsa file\u003cspan class=\"o\"\u003e]\u003c/span\u003e USER@SERVER\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eWhere \u003ccode\u003e-i\u003c/code\u003e indicates where the rsa file is located. The .pub key is the one needed to be copied.\u003c/p\u003e\n\u003cp\u003e[Optional] Start a SSH agent, and add the key to the SSH agent:\u003c/p\u003e","title":"Mounting remote drives locally using sshfs"},{"content":"Installation 1# macOS 2brew install python3 python-pip 3 4# Ubuntu 5sudo apt install python3 python-pip 6 7# Update pip to the latest version 8pip install --upgrade pip setuptools wheel 9pip3 install --upgrade pip setuptools wheel 10 11# Install virtualenvwrapper 12pip3 install virtualenv virtualenvwrapper Create projects directory 1mkdir ~/Projects Configure .profile (Bash) or .zprofile (ZSH): 1# needed for virtualenvwrapper 2export WORKON_HOME=$HOME/.virtualenvs 3export PROJECT_HOME=$HOME/Projects 4export VIRTUALENVWRAPPER_PYTHON=/bi/home/USER/.linuxbrew/bin/python3 5export VIRTUALENVWRAPPER_VIRTUALENV=/bi/home/USER/.linuxbrew/bin/virtualenv 6export PIP_REQUIRE_VIRTUALENV=true 7source /bi/home/USER/.linuxbrew/bin/virtualenvwrapper.sh Reload profile 1source ~/.profile 2source ~/.zprofile Make a new project The command will create a python virtual environment by running mkvirtualenv. Then it will creates a project directory and will turn to that directory. 1# using the default python version 2mkproject projectName 3 4# using specific python version 5mkproject -p `which python` projectName To delete a project 1rmvirtualenv projectName Bonus If you are using ZSH shell with prezto (recommended), you can change prezto prompt to indicate the virtual environment currently in use. You need to edit prompt_sorin_setup as follow1 :\n1cd $HOME/.zprezto/modules/prompt/functions/ 1# Check for activated virtualenv 2if (( $+functions[python-info] )); then 3python-info 4fi 5 6# Compute slow commands in the background. 1 \u0026#39;status\u0026#39; \u0026#39;$(coalesce \u0026#34;%b\u0026#34; \u0026#34;%p\u0026#34; \u0026#34;%c\u0026#34;)%s%A%B%S%a%d%m%r%U%u\u0026#39; 2 3# %v - virtualenv name. 4zstyle \u0026#39;:prezto:module:python:info:virtualenv\u0026#39; format \u0026#39;(%v)\u0026#39; 5 6# Define prompts. 7PROMPT=\u0026#39;${SSH_TTY:+\u0026#34;%F{9}%n%f%F{7}@%f%F{3}%m%f \u0026#34;}%F{4}$python_info[virtualenv] ${_prompt_sorin_pwd}%(!. %B%F{1}#%f%b.)${editor_info[keymap]} \u0026#39; [1] How to display current virtualenv in your ZSH Prezto theme\n","permalink":"https://firas.phd/posts/python_virtualwrapper/","summary":"\u003ch3 id=\"installation\"\u003eInstallation\u003c/h3\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 1\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# macOS\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 2\u003c/span\u003e\u003cspan class=\"cl\"\u003ebrew install python3 python-pip\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 3\u003c/span\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 4\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Ubuntu\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 5\u003c/span\u003e\u003cspan class=\"cl\"\u003esudo apt install python3 python-pip\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 6\u003c/span\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 7\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Update pip to the latest version\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 8\u003c/span\u003e\u003cspan class=\"cl\"\u003epip install --upgrade pip setuptools wheel\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e 9\u003c/span\u003e\u003cspan class=\"cl\"\u003epip3 install --upgrade pip setuptools wheel\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e10\u003c/span\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e11\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Install virtualenvwrapper\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e12\u003c/span\u003e\u003cspan class=\"cl\"\u003epip3 install virtualenv virtualenvwrapper\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch3 id=\"create-projects-directory\"\u003eCreate projects directory\u003c/h3\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e1\u003c/span\u003e\u003cspan class=\"cl\"\u003emkdir ~/Projects\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch3 id=\"configure-profile-bash-or-zprofile-zsh\"\u003eConfigure .profile (Bash) or .zprofile (ZSH):\u003c/h3\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e1\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# needed for virtualenvwrapper\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e2\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003eexport\u003c/span\u003e \u003cspan class=\"nv\"\u003eWORKON_HOME\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"nv\"\u003e$HOME\u003c/span\u003e/.virtualenvs\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e3\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003eexport\u003c/span\u003e \u003cspan class=\"nv\"\u003ePROJECT_HOME\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"nv\"\u003e$HOME\u003c/span\u003e/Projects\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e4\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003eexport\u003c/span\u003e \u003cspan class=\"nv\"\u003eVIRTUALENVWRAPPER_PYTHON\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e/bi/home/USER/.linuxbrew/bin/python3\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e5\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003eexport\u003c/span\u003e \u003cspan class=\"nv\"\u003eVIRTUALENVWRAPPER_VIRTUALENV\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e/bi/home/USER/.linuxbrew/bin/virtualenv\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e6\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003eexport\u003c/span\u003e \u003cspan class=\"nv\"\u003ePIP_REQUIRE_VIRTUALENV\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"nb\"\u003etrue\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e7\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003esource\u003c/span\u003e /bi/home/USER/.linuxbrew/bin/virtualenvwrapper.sh\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch3 id=\"reload-profile\"\u003eReload profile\u003c/h3\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e1\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003esource\u003c/span\u003e ~/.profile\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e2\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003esource\u003c/span\u003e ~/.zprofile\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch3 id=\"make-a-new-project\"\u003eMake a new project\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003eThe command will create a python virtual environment by running \u003ccode\u003emkvirtualenv\u003c/code\u003e.\u003c/li\u003e\n\u003cli\u003eThen it will creates a project directory and will turn to that directory.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e1\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# using the default python version\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e2\u003c/span\u003e\u003cspan class=\"cl\"\u003emkproject projectName\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e3\u003c/span\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e4\u003c/span\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# using specific python version\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e5\u003c/span\u003e\u003cspan class=\"cl\"\u003emkproject -p \u003cspan class=\"sb\"\u003e`\u003c/span\u003ewhich python\u003cspan class=\"sb\"\u003e`\u003c/span\u003e projectName\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch3 id=\"to-delete-a-project\"\u003eTo delete a project\u003c/h3\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e1\u003c/span\u003e\u003cspan class=\"cl\"\u003ermvirtualenv projectName\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003chr\u003e\n\u003ch3 id=\"bonus\"\u003eBonus\u003c/h3\u003e\n\u003cp\u003eIf you are using ZSH shell with prezto (recommended), you can change prezto prompt to indicate the virtual environment currently in use. You need to edit \u003ccode\u003eprompt_sorin_setup\u003c/code\u003e as follow\u003csup\u003e1\u003c/sup\u003e :\u003c/p\u003e","title":"Managing Python virtual environments"},{"content":"A colleague of mine asked me for help in using DaPars for analysing alternative polyadenylation in their RNA-seq dataset. So, I thought to write a short post here to describe how I use it.\nFrom Xia et al. 2014\nHere we develop a novel bioinformatics algorithm (DaPars) for the de novo identification of dynamic APAs from standard RNA-seq.\nInstallation Download the source files of DaPars from GitHub and extract the files:\n1$ tar zxvf DaPars-VERSION.tar.gz Input files You can find more details on the documentation page, but in essence, DaPars requires the following files:\nBED file: a tab separated, 12 columns, which represents the gene model. BedGraph file: stores the reads alignment results from an aligned BAM file. Gene Symbol file: two columns containing NCBI RefSeq and gene symbol. The BED file of the gene model can be downloaded from UCSC Table Browser.\ngenome: mouse assembly: July 2007 (NCBI37/mm9) group: Genes and Gene Predictions track: REfSeq Genes table: refGene region: genome output format: BED - browser extensible data output file: mm9_refseq_whole_gene.bed Click \u0026lsquo;get output\u0026rsquo; button, and in the next page \u0026lsquo;Output refGene as BED\u0026rsquo; click \u0026lsquo;get output\u0026rsquo; button.\nTo generate the BedGraph files from BAM files, you need the chromsome size file chromInfo.txt.gz which can be downloaded from UCSC (hg19 or mm9) and then use the BedTools\u0026rsquo; genomecov as follow:\n1$ bedtools genomecov -bg -ibam sample_sorted.bam -g mm9_chr_size.txt -split \u0026gt; sample.bedgraph Where:\n-bg : report depth in BedGraph format. -ibam : use BAM file as input for coverage. -g : the genome file. -split : Treat “split” BAM or BED12 entries as distinct BED intervals when computing coverage. The Gene Symbol file can be downloaded from UCSC Table Browser.\ngenome: mouse assembly: July 2007 (NCBI37/mm9) group: Genes and Gene Predictions track: REfSeq Genes table: refGene region: genome output format: selected fields from primary and related tables output file: mm9_30_03_2016_Refseq_id_from_UCSC.txt Click \u0026lsquo;get output\u0026rsquo; button, and in the next page select\nname: Name of gene (usually transcript_id from GTF) name2: Alternate name (e.g. gene_id from GTF) Click \u0026lsquo;get output\u0026rsquo; and save the file.\nUsage 1. Generate region annotation DaPars will use the extracted distal polyadenylation sites to infer the proximal polyadenylation sites based on the alignment files.\n1$ python DaPars_Extract_Anno.py -b mm9_refseq_whole_gene.bed -s mm9_30_03_2016_Refseq_id_from_UCSC.txt -o mm9_refseq_extracted_3UTR.bed Where:\n-b GENE_BED_FILE : The gene model in BED format -s Gene_Symbol_FILE : The mapping of transcripts to gene symbol, which can be extracted from UCSC Tables. -o OUTPUT_FILE : The output of the extracted annotation region. The structur of the DaPars folder looks like this:\n1. 2├── DaPars_Extract_Anno.py 3├── DaPars_main.py 4├── mm9_30_03_2016_Refseq_id_from_UCSC.txt 5├── mm9_chr_size_chromInfo.txt 6└── mm9_refseq_whole_gene.bed Since I am using Sun Grid Engine (SGE), I used the following job script to perform this step\n1#!/bin/sh 2module load bedtools 3 4PROJECT_DIR=$HOME/Projects/RNAseq_Project 5DAPARS=$HOME/dapars/ 6CHROMINFO=$HOME/dapars/mm9_chr_size_chromInfo.txt # The extracted chromInfo.txt.gz 7MM9_REFSEQID=$HOME/dapars/mm9_30_03_2016_Refseq_id_from_UCSC.txt 8MM9_BED=$HOME/dapars/mm9_refseq_whole_gene.bed 9 10 11cat \u0026gt; $PROJECT_DIR/Region-Annotation.sh \u0026lt;\u0026lt;EOF 12python $DAPARS/DaPars_Extract_Anno.py -b $MM9_BED -s $MM9_REFSEQID -o mm9_refseq_extracted_3UTR.bed 13EOF 14 15qsub -l h_vmem=2G -V -cwd -m e -M email@example.com $PROJECT_DIR/Region-Annotation.sh 2. Sample processing The files generated in step 1 above will be used in step 2. Also for this step, you need to generate configure_file for each sample. For example:\n1#The following file is the result of step 1. 2Annotated_3UTR=mm9_refseq_extracted_3UTR.bed 3 4#A comma-separated list of BedGraph files of samples from condition 1 5Group1_Tophat_aligned_Wig=KO1.bedgraph,KO2.bedgraph,KO3.bedgraph 6 7#A comma-separated list of BedGraph files of samples from condition 2 8Group2_Tophat_aligned_Wig=WT1.bedgraph,WT2.bedgraph,WT3.bedgraph 9 10Output_directory=DaPars_Output_Condition1/ 11Output_result_file=DaPars_Output_Condition1 12 13#At least how many samples passing the coverage threshold in two conditions 14Num_least_in_group1=1 15Num_least_in_group2=1 16 17Coverage_cutoff=30 18 19#Cutoff for FDR of P-values from Fisher exact test. 20 21FDR_cutoff=0.05 22PDUI_cutoff=0.5 23Fold_change_cutoff=0.59 The following is the SGE job script that I used to perform this step.\n1#!/bin/sh 2module load bedtools 3 4PROJECT_DIR=$HOME/Projects/RNAseq_Project 5DAPARS_SOURCE=$HOME/dapars 6DAPARS_CONFIG_FILE=$HOME/dapars/configure_file 7 8GROUP01=Cell_type_01 9GROUP02=Cell_type_02 10GROUP03=Cell_type_03 11GROUP04=Cell_type_04 12 13for G_ID in {01..04} #Where G_ID is Group ID 14do 15 export GROUP=GROUP${G_ID} 16 export GROUP_ID=${!GROUP} 17cat \u0026gt; $PROJECT_DIR/DaPars_${GROUP_ID}.sh \u0026lt;\u0026lt;EOF 18python $DAPARS/DaPars_main.py configure_file_${GROUP_ID} 19EOF 20 21qsub -l h_vmem=2G -V -cwd -m e -M email@example.com $PROJECT_DIR/DaPars_${GROUP_ID}.sh 22 23done ","permalink":"https://firas.phd/posts/dapars/","summary":"\u003cp\u003eA colleague of mine asked me for help in using \u003ca href=\"https://github.com/FirasSadiyah/dapars\"\u003eDaPars\u003c/a\u003e for analysing alternative polyadenylation in their RNA-seq dataset. So, I thought to write a short post here to describe how I use it.\u003c/p\u003e\n\u003cp\u003eFrom \u003ca href=\"http://www.nature.com/articles/ncomms6274\"\u003eXia \u003cem\u003eet al\u003c/em\u003e. 2014\u003c/a\u003e\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003eHere we develop a novel bioinformatics algorithm (DaPars) for the de novo identification of dynamic APAs from standard RNA-seq.\u003c/p\u003e\u003c/blockquote\u003e\n\u003ch1 id=\"installation\"\u003eInstallation\u003c/h1\u003e\n\u003cp\u003eDownload the source files of DaPars from \u003ca href=\"https://github.com/ZhengXia/dapars\"\u003eGitHub\u003c/a\u003e and extract the files:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"ln\"\u003e1\u003c/span\u003e\u003cspan class=\"cl\"\u003e$ tar zxvf DaPars-VERSION.tar.gz\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch1 id=\"input-files\"\u003eInput files\u003c/h1\u003e\n\u003cp\u003eYou can find more details on the \u003ca href=\"http://lilab.research.bcm.edu/dldcc-web/lilab/zheng/DaPars_Documentation/html/DaPars.html\"\u003edocumentation page\u003c/a\u003e, but in essence, DaPars requires the following files:\u003c/p\u003e","title":"DaPars for alternative polyadenylation analysis"}]