Prepro: Embedded Perl/Python preprocessor
© Copyright 2007 Adrian Lewis
 

Features
Download
Introduction
Why
Theory
Running

   

Prepro Home Page

Prepro is an embedded Perl/Python preprocessor for text files, e.g. Verilog source code. Prepro allows Perl or Python to be embedded in a text file in a similar manner to a standard preprocessor, with preprocessor statements and variables, but with the functionality of a full-featured scripting language.

Features

  • Power preprocessing
    • All the features of Perl or Python.
    • Multiple builds from the same source file.
    • Pre-processor variable and expression substitutions.
  • Ease of use
    • Pre-processor statements easy to use and add to existing text files.
    • Single or multi-line embedded preprocessor code.
    • Reconfigurable keywords allow embedding into any line based text file.
  • Simple installation: the prepro executable is a single Python file.
  • Debuggable: intermediate Perl/Python file output.
  • Fifofum: A Prepro-based Verilog FIFO generator is bundled with Prepro.
  • Open source: standard GNU License, no sign-up required, no special conditions of use!

Download

LATEST SOURCE CODE prepro-2.1.tar.gz
VERSION 2.1
DATE POSTED Feb 19th 2007
Release Notes Added support for substitutions with padding to the original text width.

Introduction

Source text is often composed of chunks of repetitive code. The following example shows a fragment of Verilog RTL used to reverse the order of bits in a 4-bit bus.
  out[ 3 ] = in[ 0 ];
  out[ 2 ] = in[ 1 ];
  out[ 1 ] = in[ 2 ];
  out[ 0 ] = in[ 3 ];
It's easy to make typographical errors and difficult to extend the width of the bus to 256 bits without lots of error-prone typing. Now let's try to generate this code using prepro and Perl.
// pl for $bit (0..$width-1) {
  out[ #[$width-$bit-1]# ] = in[ #$bit ];
// pl }
Now run: prepro file -d '$width = 4;' to obtain the same output as the original text.

Lines starting with "// pl" are preprocessor lines coded in Perl, in this case a for-loop. The reference "#$bit" substitutes the value of the Perl variable "$bit" into the output text. The reference "#[$width-$bit-1]#" substitutes the value of the Perl expression "$width-$bit-1" into the output text. The value of the Perl variable "$width" is passed into prepro using the command line argument "-d '$width = 4;'".

This is a very simple example and could be recoded entirely Verilog, but for more complex outputs Verilog and even SystemVerilog quickly run out of stream.

Why use prepro?

What I'm currently doing works pretty well; why should I change?
  • Repetitive text is error-prone, inflexible, and hard to maintain. Most languages have facilities for generating repetitive code but there are often annoying limitations. Using cut-and-paste to generate repetitive code by hand is prone to typographical errors and is hard to maintain and verify. Prepro makes it easy to generate repetitive code that's easier to maintain and is typo-free.
  • Similar code chunks. How should I code similar but not identical code chunks? One of the trickiest decisions to make when coding is whether to maintain two separate but similar chunks of code or whether to combine the chunks into a single source and add run-time or compile-time switches to modify the behaviour of each instance. The decision depends on the ratio of differences to similarities, and the overhead of adding switches. Prepro provides powerful preprocessing facilities, making it easier to keep a single maintainable source file.
  • Power of scripting languages, weakness of standard preprocessors. Some languages provide a built-in preprocessor, but often it has severe limitations. Using Perl or Python as the preprocessing language, the preprocessor has access to libraries, high-level structures, and all the other advanced features of these scripting languages.
  • Multiple builds. Standard preprocessors use global static constants to decide which code chucks are included in the compiled output. As the decision is based on a constant, only one version of the post-processed code is available. Prepro can be used to produce multiple outputs from the same input code, so all or any versions of the code are available.
  • Debug and code coverage. As Prepro can remove all unnecessary text from its output, using this text as the input to debug and code-coverage makes these tasks simpler. Ignoring unused code when debugging or assessing code-coverage results is no longer required.
  • Reconfigurable code. Building complex systems can be challenging. Adding reconfigurability to the build provides invaluable flexibility, but it's not often the sort of task that a standard preprocessor can handle.

Theory of operation

Prepro begins by processing the command line arguments. Prepro then reads an input file or the standard input line by line and generates an output Perl or Python script file. Lines from the input are either lines of preprocessor code or source text. The lines of preprocessor code are written to the output script file, lines of source text are converted into print statements. If the line of source text contains variable or expression substitutions, the print statement contains references to the corresponding variable or expression. Once the entire output script file has been generated, Prepro executes the script redirecting the output to the output file name specified on the prepro command line. If the script executed without error (and the "-k" option was not set), the intermediate script file is deleted.

Running Prepro

Prepro is a Python script that is typically run as part of the "make" build process. Prepro reads an input file or stream and outputs to a file or standard output. There are a number of command line options which fall into the following categories:
  • Set input and output file names.
  • Including preprocessor code from the command line.
  • Select Perl or Python preprocessor.
  • Debugging the intermediate script.
  • Customizing Prepro key-words.
Usage: prepro [OPTION]... [FILE]
run Perl or Python preprocessor on a text file.

           FILE          Name of the input text file, standard input "-" by default.

  -o FILE, --output FILE Name of the output text file, standard output "-" by default.

  -d LINE, --define LINE Line of preprocessor code inserted before the preprocessor code
                         generated from the input. This option is typically used to define
                         preprocessor variables from the command line.

  -h,      --help        Print this usage information. Quit preprocessor.

  -pl,     --perl        Select the Perl preprocessor (set by default); set Perl defaults.

  -py,     --python      Select the Python preprocessor; set Python defaults.

  -f FILE, --file FILE   Name of intermediate file generated by prepro.
                         Default: .<output_file_name> or .prepro if the output is the
                         standard output "-"

  -k,      --keep        Do not delete the intermediate file.

  -c ID,   --code ID     Identifier for preprocessor code
                         Default usage for Perl:   // pl <perl code>
                         Default usage for Python: // py <python code>
                         Note that white-space before // is deleted.

  -b ID,   --begin ID    Identifier for beginning of multi-line preprocessor code
                         Default for Perl:   /* pl-begin
                         Default for Python: /* py-begin

  -e ID,   --end ID      Identifier for end of multi-line preprocessor code
                         Default for Perl:   pl-end */
                         Default for Python: py-end */

  -r ID,   --replace ID  Text to be replaced with a preprocessor variable or expression.
                         Defaults: #$ and #[;]# respectively
                         Occurrances of #$<variable> in the input will be replaced with
                         the value of the preprocessor variable in the output.
                         Occurrances of #$<variable>#text in the input will be replaced
                         with the value of the preprocessor variable in the output and the
                         delimiting # removed.
                         Occurrances of #[<expression>]# in the input will be replaced with
                         the value of the preprocessor expression in the output.
                         A semi-colon denotes an expression substitution and the location
                         of the expression.

  -rl ID,  --left ID     Identical to -r/--replace except the output text is padded with
                         trailing spaces to the width of the input text.
                         Default: #<;<#

  -rr ID,  --right ID    Identical to -r/--replace except the output text is padded with
                         leading spaces to the width of the input text.
                         Default: #>;>#

  ++ ARGS                Command line arguments may be passed to the preprocessor script
                         using ++ followed by a list of arguments.

This website uses CSS stylesheets and Frames.