A beginning user manual for clj-peg
Contents
1. bbb as a String This PEG accepts a string with one or more b s and nothing else make parser main simple parser ACIS This is a simple grammar SpIS ul Sama A lt gt i B B s igi p Listing 1 A simple PEG defined in clj peg When the newly created functions have finished process ing the input bbb they return a structure that describes the input and this is traditionally referred to as an AST Abstract Syntax Tree This result of parsing the input is shown in List ing 2 You can see that the data structures used in the result closely match the data structures used in the grammar The way the macro make parser works allows you to treat a grammar definition like any other data structure If you share your grammar I d recommend simply tying it to some symbol using def and allowing those who use it to eval the related symbol This gives your users the opportunity of tweaking your grammar to fit their needs There are more details about the process of tweaking a grammar in later sec tions l Note that I m making a distinction between input pattern and input type While the pattern of input that can be processed is defined as shown in Listing 1 the type is not necessarily defined Specifically clj peg doesn t care what type of input you provide since it must be wrapped in a specific type of struct That specific type of struct acts as an interface between the generated parsing code and some input2. still open to feedback but at version 0 6 the details are starting to solidify Please contact me if you have anything that I can help with While I don t pretend to have as much time as I m filling I do try to care Good luck and don t be afraid to share your grammars
3. A beginning user manual for clj peg Richard Lyman www lithinos com This documentation was written for version 0 6 This manual is written for those who are comfortable de veloping in Clojure and who may or may not be comfortable with parsing lexing or grammar construction You should not need to have an advanced degree to understand this man ual I have written each section as tightly as I could provid ing for shorter and more numerous sections If you re un sure where to start simply start reading from the very begin ning and continue Each section will flow into the next and should progress nicely If there are parts where my explana tions leave you confused please let me know If you re familiar with terms such as AST or non greedy consumption you should skip along on the section titles and skim the code until you find something new If there are parts that you find ambiguous please let me know Either way I hope this manual helps you to get up to speed as quickly as you want I also hope that you become effec tive in using the clj peg library and that that effectiveness will encourage you to share your PEGs with others Overview When you create a PEG you create a set of functions that are able to parse input The specific pattern of input that those functions parse is defined in the rules of the PEG you create Starting with a PEG as defined in Listing 1 we can use the functions it creates to process the input
4. answer Expr interpreter ast printin input answer Listing 5 A simple example Advanced Example Let s extend the simple example with a few changes We ll be allowing multiplication and division and we ll do so ina way that preserves the required precedence There isn t any thing too difficult with these changes We ll also be expanding the definition of what numbers are allowed by re using part of a JSON grammar This change is important since it can show how simple it is to mix and match parts of grammars You can notice that the changes to re use part of a JSON grammar are even easier than the changes to allow multiplication and division ns com company project main gen class use com lithinos clj peg core string peg json parser json interpreter def math grammar make parser main math doc The characters and are the only operators and JSONNumbers are the only operands srules Expr lt Sum Sum lt Product x S mOp Product y Product lt Value Prod ct0op Value Value lt Num Expr Num lt JSONNumber SumOp Ls Productors lt 37 1s8 BY declare Expr interpreter defn Num interpreter ast JSONNumber interpreter ast Num defn Value interpreter ast let ast ast Value if contains ast Expr Expr interpreter ast Num interpreter ast defn product reduction a b let temp op first b Produc
5. e we would not use gen class or have a main method or call eval on the grammar Expr fesun Valus Num TERI esSumeye Var fewailwe fein We iy fssSimey Y aede sini SM 1 Empty true IB Listing 4 The AST from the grammar in Listing 5 applied to 142 3 won t bother with Empty since in this case it doesn t matter The non terminals sumop and Num are easily handled inside Value and Sum The complete code is shown in Listing 5 ns com company project main gen class use com lithinos clj peg core string peg def g make parser main math doc The characters and are the only operators and positive whole integers are the only operands srules Expr lt Sum i Sum lt Value SumOp Value Value lt Num Expr Num 58 OSA sypualojey lt e 338 far 4 Hy declare Expr interpreter defn Value interpreter ast let ast ast Value Gli Wicontarns aS EXPE Expr interpreter ast Integer parseInt ast Num defn Sum interpreter ast let ast ast Sum reduce fm ab let water first bD sSumOp rand Value interpreter second b 22 rater m a rand Cand Value interpreter first ast second ast defn Expr interpreter ast Sum interpreter first ast Expr eval g defn main amp args lee imyemtie Wile 2 5 valid struct create string input input ast math valid struct
6. r is the extension of our math grammar to include JsONNumbers Even though the im plementation of the JSONNumber interpreter is not com plete our simple use of it in this case will still work For the first step we need to import the json interpreter and json parser namespaces Second we reference the JSONNumber non terminal in our math grammar Third we evaluate the json grammar before we evaluate our gram mar so that the definition for JsoNNumber is available when we evaluate our grammar Fourth and finally we call the JSONNumber interpreter inside our Num interpreter Conclusion I need to write another manual This one does not cover everything You ve likely noticed that there were things that I didn t explain such as the create string input function In ad dition there are several parts of the clj peg library that aren t finalized in my mind These will all require at least one more manual For now this manual is sufficient to scratch my documen tation itch I have too many projects all running along at the same time and this is just one of them It was bumped to the front of the queue since I m using it in other projects that were themselves at the front There are some issues that I want to address before the 0 8 and 1 0 releases but I m hoping that they re all internal As a consequence of that I m also hoping that this documentation will only minimally change as clj peg moves toward a 1 0 release I m
7. re are a few explanations that come first In any clj peg definition the three items to notice are the main func tion the doc string and the rules The main function is set to math which is what you ll call to use the parser The doc string is what accompanies the resulting main function The rules commonly referred to as productions define the differ ent non terminals and terminals as well as how they re joined to form all of the possible combinations of input that can be parsed 2Depending on your intended use you may need to purchase a license Sequence A B The parser must consume A first and that must be followed by con suming B Ordered choice A B The parser must either consume A or B and the parser must try A first followed by trying B Zero or more A The parser must greedily consume zero or more of A One or more A The parser must greedily consume at least one and will greedily con sume more than one of A if it can One or none A The parser must consume one of A if it can And predicate amp A The parser must be able to con sume A and must not consume A Not predicate A The parser must not be able to con sume A Table 1 Operators and delimiters for combining non terminals and terminals in a clj peg definition Rules in clj peg use operators and delimiters to combine non terminals If you are familiar with PEG grammars in gen eral you ll find that these fi
8. t the basic definition All of the provided combinations are shown in Table 1 These delim iters and operators follow Clojure data structures and function calls Simple Example We re going to be developing a grammar that parses simple mathematical operations For now we ll only support positive whole integers as operands and we ll only support addition and subtraction as operators The entire result can be seen in Listing 5 so let s look at the grammar tied to the symbol g In this specific definition the non terminals are Expr Sum and value and the terminals are Num and Sumop Notice use of the Clojure reader macro for Patterns in the terminals Num and sumop Currently non terminals can be anything that is seqable but is not a String the rest can be used in terminals Writing an interpreter for this simple grammar is fairly easy Each non terminal that is part of an AST has the form Non Terminal lt body gt There is only one key value pair in the map the key will be based on the non terminal and the body will match the grammar in structure and use of RHS combinations Let s look at the result from the grammar in Listing 5 and apply it to the expression 1 2 3 The result ing AST is shown in Listing 4 Interpreting this AST can be approached by defining a function for selected non terminals We ll define functions for the non terminal mappings Expr Sum and value We 3 If we were to distribute this cod
9. tOp operator if temp op operandl a operand2 Value interpreter second b operator operandl operandz2 defn Product interpreter ast let ast ast Product if contains second ast Zero or more Value interpreter first ast reduce product reduction Value interpreter first ast second ast defn sum reduction a b let temp op first b SumOp operator if temp op operandl a operand2 Product interpreter second b operator operandl operandz2 defn Sum interpreter ast let ast ast Sum if contains second ast Zero or more Product interpreter first ast reduce sum reduction Product interpreter first ast second ast defn math interpreter ast Sum interpreter first ast Expr eval json grammar eval math grammar defn main amp args Gliets input rea ou valid struct create string input input ast math valid struct answer math interpreter ast printin input answer Listing 6 An advanced example The additions for multiplication and division are sim ple The non terminal Product becomes similar to what sum was and placing Product below sum in the AST allows our bottom up interpreter the opportunity of solving products be fore sums You can see as well that the interpreter for the new Product non terminal is very similar to the interpreter for Sum The interesting part howeve
10. type B b 2B b B b Empty true Listing 2 The result of applying the parser generated from the grammar in Listing on the input bbb Setup To start getting our hands dirty with clj peg we ll need to install it There really isn t much to installing clj peg just download it extract it and put the JAR file on your class path For a more complete walkthrough on setting up a Clo jure project I wrote a post with details on my typical setup Using that typical setup I d place the clj peg jar file in the dst lib folder After the clj peg JAR is on the classpath you can use the library in several ways For this manual we ll be including code through the use reference in the ns macro Our main file will look like Listing 3 to begin with ns com company project main gen class use com lithinos clj peg Goes string peg defn main amp args printin Works Listing 3 Using clj peg code For help with the commands to compile and run a Clo jure application I d again refer you to my post with details on my typical setup that includes very simple Ant scripts You should easily be able to understand what needs to happen when compiling or running a Clojure application and should easily be able to integrate that process into your own build approach if you choose to not use those Ant scripts Examples I m going to provide some examples in the following sec tions but the
Download Pdf Manuals
Related Search