Stateful parsing with Megaparsec

2026-04-20T00:00:00+00:00
Introduction</h1>
Recently, I’ve been learning about writing parsers using megaparsec</a>. I discovered a wonderfully comprehensive guide</a> by Mark Karpov on this matter.</p>
Something that I wanted to implement in my parser was the ability to “remember” and “forget” information, essentially for the parser to have some internal state. I found the tutorial’s explanation on this a bit confusing, I’m hoping this tutorial makes this a bit clearer for anyone else trying to something similar.</p>
Over the course of this tutorial, we will be making a simple chatbot that manages a to-do list. Here’s a sneak peek:</p>
</p>
Whilst this chatbot is quite simple, I have implemented a much more capable and versatile chatbot here</a>, which uses a similar stateful parser at its core. That challenges that I faced developing that led to me write this blog post!</p> </blockquote>
Getting started</h1>
We can begin building our Megaparsec parser using the Parsec</code> type. This type has quite a few parameters, and it looks (somewhat) like this:</p>
data Parsec</span> e s a</span> =</span> Parsec</span> e s a</span></span></code></pre>
Here, e</code> is the custom error component type. This could be useful for more complicated parsers, but for now, we can use skip the hassle and use Void</code> indicating that our parser doesn’t define any custom errors. s</code> is the input stream type. Since we are working with a simple stream of characters, String</code> would be a suitable choice. Finally, a</code> is the return type of the parser.</p>
Instead of having to incant this obtuse type everywhere, its much more common to define a type synonym like the following. We haven’t specified the return type just yet, this will allow for some flexibility later.</p>
import</span> Text.Megaparsec</span></span>
import</span> Data.Void</span> (</span>Void</span>)</span></span>
</span>
-- NOTE: `Parser` is of kind * -> *</span></span>
type Parser = Parsec Void String</span></span></code></pre>
Now that we defined a type for our parser, let’s now write our first parser. We don’t necessarily have to implement everything</em> from scratch: Text.Megaparsec</code>, Text.Megaparsec.Char</code>, and Text.Megaparsec.Char.Lexer</code> provide some useful building blocks for us to get started with. Here is an example:</p>
import</span> Text.Megaparsec.Char</span></span>
</span>
parseHello</span> :: Parser</span> ()</span></span>
parseHello </span>= do</span></span>
  string </span>"hello"</span></span>
  pure </span>()</span></span></code></pre>
The type signature for our parser makes it clear what gets returned after a successful parse. In this case, it is the unit type. All that parseHello</code> does is parse in the string “hello”, and then return the unit type.</p>
Returning the unit type doesn’t really tell us much about what was just parsed. Our ultimate goal is to parse in some user request, and return some response. Let’s define a custom datatype, Response</code>, and then create various constructors for all the possible responses that can be given by our program.</p>
import</span> Data.List</span> (</span>intercalate</span>)</span></span>
</span>
data Response =</span> Remember</span> |</span> Complete</span> |</span> List</span> [</span>String</span>]</span></span>
</span>
instance Show Response where</span></span>
  show </span>Remember</span> =</span> "Okay, I'll remember that."</span></span>
  show </span>Complete</span> =</span> "Okay, I'll mark that as completed."</span></span>
  show (</span>List</span> tasks)</span> =</span> intercalate </span>"</span>\n</span>"</span> $</span> map (</span>" - "</span> ++</span>) tasks</span></span>
</span>
parseRemember</span> :: Parser Response</span></span>
parseRemember </span>= do</span></span>
  string </span>"remember to"</span></span>
  pure </span>Remember</span></span>
</span>
parseComplete</span> :: Parser Response</span></span>
parseComplete </span>= do</span></span>
  string </span>"mark as completed"</span></span>
  pure </span>Complete</span></span>
</span>
parseList</span> :: Parser Response</span></span>
parseList </span>= do</span></span>
  string </span>"list tasks"</span></span>
  pure </span>$</span> List []</span> -- Returning an empty list for now</span></span></code></pre>
I have added 3 new parsers here, each of which focuses on parsing in a specific type of user request, and returning the appropriate Response</code> value. The Show</code> instance for Response</code> defines how the program should format the response when it gets printed onto the terminal.</p>
Structuring parsers like this makes them read somewhat like a BNF grammar</a>! It also helps us separate our concerns: each parser only has to worry about dealing with that specific request.</p>
To combine these parsers together, we can use the alternative (<|></code>) operator. In the context of parsing, a <|> b</code> would try running parser a</code> first, and if it fails, it tries running b</code>. The choice</code> combinator also achieves something similar, but let’s stick with the alternative operator for now:</p>
parseRequest</span> :: Parser Response</span></span>
parseRequest </span>=</span> parseRemember </span><|></span> parseComplete </span><|></span> parseList</span></span></code></pre>
We’ve written quite a few parsers by now, but we haven’t actually run anything yet. To run parsers of type Parsec</code> we can use the runParser</code> function (synonymous to parse</code>). It takes 3 arguments, the parser, the filename (not super important here, can be left blank), and the actual string we want to parse.</p>
runParser parseRequest </span>"<stdin>" "list tasks"</span></span></code></pre>
This returns an Either</code> type, which is either a ParseErrorBundle</code>, or our parser’s return type when the input was parsed successfully. Since our application is quite simple, we aren’t going to be handling errors in depth. The errorBundlePretty</code> function could be useful here if we were to do that: it converts an error bundle to a string that can be printed.</p>
We can put this in a loop to create the REPL for our program.</p>
import</span> Control.Monad</span> (</span>forever</span>)</span></span>
import</span> System.IO</span> (</span>hSetBuffering</span>,</span> stdout</span>,</span> BufferMode</span>(..))</span></span>
</span>
loop</span> :: IO</span> ()</span></span>
loop </span>=</span> forever </span>$ do</span></span>
  putStr </span>"> "</span></span>
  request </span><-</span> getLine</span></span>
  </span></span>
  let</span> eitherResponse </span>=</span> runParser parseRequest </span>"<stdin>"</span> request</span></span>
  </span></span>
  putStrLn </span>$ case</span> eitherResponse </span>of</span></span>
    Left</span> _ </span>-></span> "Sorry, I didn't understand that."</span></span>
    Right</span> response </span>-></span> show response</span></span>
</span>
main</span> :: IO</span> ()</span></span>
main </span>= do</span></span>
  hSetBuffering stdout </span>NoBuffering</span></span>
  loop</span></span></code></pre>
The program should now be able to prompt the user for a string, and when one is provided, it will parse it. If any parse errors occur, and error message will be printed. If a request is successfully parsed, we format and convert the response to a string and print it. This logic loops over and over again.</p>
At the moment, users cannot specify the names of the tasks they want our program to remember. To implement this, we must be able to define some sort of parser that can take in arbitrary strings. We’ll accept strings surrounded by quotes, this makes parsing easy: the quotes make it clear where the string starts and ends. Here is a parser that does that:</p>
parseQuoted</span> :: Parser String</span></span>
parseQuoted </span>=</span> between (char </span>'"'</span>) (char </span>'"'</span>) (some </span>$</span> anySingleBut </span>'"'</span>)</span></span></code></pre>
The between</code> combinator does exactly what it says; the first two arguments specify the starting and closing quotes, and the third argument is the string we want to return. Our string is a series of characters (except "</code>). The some</code> combinator specifies that our string can be arbitrarily long but must have atleast 1 character.</p>
Now adding this to our request parsers:</p>
parseRemember</span> :: Parser Response</span></span>
parseRemember </span>= do</span></span>
  string </span>"remember to "</span></span>
  task </span><-</span> parseQuoted</span></span>
  </span></span>
  pure </span>Remember</span></span>
</span>
parseComplete</span> :: Parser Response</span></span>
parseComplete </span>= do</span></span>
  string </span>"mark "</span> </span></span>
  task </span><-</span> parseQuoted</span></span>
  string </span>" as completed"</span></span>
  </span></span>
  pure </span>Complete</span></span></code></pre>
That’s it for the introduction! So far, we implemented the REPL and the core parsing logic. This introduction is in no way a comprehensive introduction to megaparsec; once again, I strongly recommend that you check out Mark Karpov’s guide</a> if you wish to learn more about the various inbuilt parsers and combinators. Now that we have gotten the fundamentals out of the way, we can focus on nitty gritty bit: implementing memory.</p>
States</h1>
Currently the loop calls the exact same stateless logic every single time, so unsurprisingly, our chatbot cannot yet remember the tasks we ask it to remember.</p>
One way we could address this is if our parsers could somehow accept the current state of its memory, say some type s</code>, and return a modified version of it alongside the existing return type a</code>. Our parsers would essentially act as state machines, returning not only the parsed data but some state data alongside that; this is analogous to the function s -> (a, s)</code>. Our loop</code> function would also need to be modified such that it calls itself with the correct state value every single time we parse something.</p>
Implementing this ourselves seems quite tedious, is there any easier way to do this?</p>
Those of you acquainted with the mtl</a> might have already spotted this, but this idea of a state machine has already been implemented via the State monad</a>! In fact, the s -> (a, s)</code> function we described is exactly what the runState</code> does.</p>
A small lie</h2>
Previously, I stated that the Parsec</code> is defined as the following:</p>
data Parsec</span> e s a</span> =</span> Parsec</span> e s a</span></span></code></pre>
However, this is not the case. Parsec</code> is actually defined as a type synonym for ParsecT</code>.</p>
type Parsec</span> e s a</span> = ParsecT</span> e s</span> Identity</span> a</span></span></code></pre>
ParsecT</code> seems quite similar to Parsec</code>, with the addition of an extra type parameter m</code>. As the suffix in its name might have indicated, ParsecT</code> is actually a monad transformer; the extra parameter m</code> is the underlying monad. In the case of Parsec</code>, the underlying monad is Identity</code>.</p>
This discovery is a crucial one: since our parsers are transformers, they can access the powers of the monads that they are layered on top off. This means that we can add a State</code> monad to our parser type, and we should get a type than can theoretically do parsing and keep track of some custom state.</p>
-- NOTE: `Parser` is of kind * -> *</span></span>
type Parser = Parsec Void String</span> (</span>State Memory</span>)</span></span></code></pre>
Here we are defining our custom state type to be Memory</code>, we haven’t yet implemented such a type so let us do that really quick.</p>
import</span> Control.Monad.State</span></span>
import</span> Text.Megaparsec</span> hiding</span> (</span>State</span>)</span></span>
</span>
newtype Memory =</span> Memory</span> [</span>String</span>]</span></span>
</span>
initialMemory</span> :: Memory</span></span>
initialMemory </span>=</span> Memory []</span></span>
</span>
remember</span> ::</span> (</span>MonadState Memory</span> m)</span> => String -></span> m</span> ()</span></span>
remember task </span>=</span> modify remember'</span></span>
  where</span></span>
    remember' (</span>Memory</span> memory)</span> =</span> Memory</span> $</span> task </span>:</span> filter (</span>/=</span> task) memory</span></span>
</span>
forget</span> ::</span> (</span>MonadState Memory</span> m)</span> => String -></span> m</span> ()</span></span>
forget task </span>=</span> modify forget'</span></span>
  where</span></span>
    forget' (</span>Memory</span> memory)</span> =</span> Memory</span> $</span> filter (</span>/=</span> task) memory</span></span>
</span>
recall</span> ::</span> (</span>MonadState Memory</span> m)</span> =></span> m [</span>String</span>]</span></span>
recall </span>= do</span></span>
  (</span>Memory</span> memory)</span> <-</span> get</span></span>
  pure memory</span></span>
</span></code></pre>
This custom memory type is just a thin wrapper over a string list. We have defined an initial state, initial</code>, which is an empty list. This represents the state the chatbot starts at when it has 0 recorded tasks. remember</code>, forget</code>, and recall</code> are all state actions to interact with the state monad, which does so via the get</code> and modify</code> functions. The types for them look a bit funky, but they allow interactions with the state monad from some other monad higher up in our transformer stack.</p>
Remembering stuff</h1>
Its quite simple to use the state actions we just implemented, we can treat them the same way we treat parser monads within do blocks. Adding these to the parsers we wrote earlier:</p>
parseRemember</span> :: Parser Response</span></span>
parseRemember </span>= do</span></span>
  string </span>"remember to "</span></span>
  task </span><-</span> parseQuoted</span></span>
</span>
  remember task</span></span>
</span>
  pure </span>Remember</span></span>
</span>
parseComplete</span> :: Parser Response</span></span>
parseComplete </span>= do</span></span>
  string </span>"mark "</span></span>
  task </span><-</span> parseQuoted</span></span>
  string </span>" as completed"</span></span>
</span>
  forget task</span></span>
</span>
  pure </span>Complete</span></span>
</span>
parseList</span> :: Parser Response</span></span>
parseList </span>= do</span></span>
  string </span>"list tasks"</span></span>
</span>
  tasks </span><-</span> recall</span></span>
</span>
  pure </span>$</span> List</span> tasks</span></span></code></pre>
It seems like we are finally done with our chatbot right? Well not quite. Thanks to how monad transformers work, all the parser functions we have written should work as is. What does break is our loop</code> code.</p>
We first update loop so that it gets called with an argument for the current memory state.</p>
loop</span> :: Memory -> IO</span> ()</span></span>
loop memory </span>= do</span></span>
  -- Parsing and state logic...</span></span>
  loop memory</span></span>
</span>
main</span> :: IO</span> ()</span></span>
main </span>= do</span></span>
  hSetBuffering stdout </span>NoBuffering</span></span>
  loop initialMemory</span></span></code></pre>
Now to run our parser, we need to use runParserT</code> instead of runParser</code> since we switched to using ParsecT</code>. However, this won’t return eitherResponse</code> like before, but rather a state action. This is because transformer stacks are like onions, where we need to peel it layer by layer from the outside inwards. runParserT</code> merely runs the outermost layer of our stack, that is, ParsecT</code>. To get the response, we need to run the inner state action with runState</code>, which returns both our desired response as well as the new memory state.</p>
loop</span> :: Memory -> IO</span> ()</span></span>
loop memory </span>= do</span></span>
  putStr </span>"> "</span></span>
  request </span><-</span> getLine</span></span>
</span>
  let</span> stateAction </span>=</span> runParserT parseRequest </span>"<stdin>"</span> request</span></span>
  let</span> (eitherResponse, newMemory)</span> =</span> runState stateAction memory</span></span>
</span>
  putStrLn </span>$ case</span> eitherResponse </span>of</span></span>
    Left</span> _ </span>-></span> "Sorry, I didn't understand that."</span></span>
    Right</span> response </span>-></span> show response</span></span>
</span>
  loop newMemory</span></span></code></pre>
And that’s it! The chatbot should fully be working as initially demonstrated!</p>

Full source code</h1>
import</span> Control.Monad.State</span></span>
import</span> Data.List</span> (</span>intercalate</span>)</span></span>
import</span> Data.Void</span> (</span>Void</span>)</span></span>
import</span> System.IO</span> (</span>hSetBuffering</span>,</span> stdout</span>,</span> BufferMode</span>(..))</span></span>
import</span> Text.Megaparsec</span> hiding</span> (</span>State</span>)</span></span>
import</span> Text.Megaparsec.Char</span></span>
</span>
newtype Memory =</span> Memory</span> [</span>String</span>]</span></span>
</span>
initialMemory</span> :: Memory</span></span>
initialMemory </span>=</span> Memory []</span></span>
</span>
remember</span> ::</span> (</span>MonadState Memory</span> m)</span> => String -></span> m</span> ()</span></span>
remember task </span>=</span> modify remember'</span></span>
  where</span></span>
    remember' (</span>Memory</span> memory)</span> =</span> Memory</span> $</span> task </span>:</span> filter (</span>/=</span> task) memory</span></span>
</span>
forget</span> ::</span> (</span>MonadState Memory</span> m)</span> => String -></span> m</span> ()</span></span>
forget task </span>=</span> modify forget'</span></span>
  where</span></span>
    forget' (</span>Memory</span> memory)</span> =</span> Memory</span> $</span> filter (</span>/=</span> task) memory</span></span>
</span>
recall</span> ::</span> (</span>MonadState Memory</span> m)</span> =></span> m [</span>String</span>]</span></span>
recall </span>= do</span></span>
  (</span>Memory</span> memory)</span> <-</span> get</span></span>
  pure memory</span></span>
</span>
type Parser = ParsecT Void String</span> (</span>State Memory</span>)</span></span>
</span>
data Response =</span> Remember</span> |</span> Complete</span> |</span> List</span> [</span>String</span>]</span></span>
</span>
instance Show Response where</span></span>
  show </span>Remember</span> =</span> "Okay, I'll remember that."</span></span>
  show </span>Complete</span> =</span> "Okay, I'll mark that as completed."</span></span>
  show (</span>List</span> tasks)</span> =</span> intercalate </span>"</span>\n</span>"</span> $</span> map (</span>" - "</span> ++</span>) tasks</span></span>
</span>
parseQuoted</span> :: Parser String</span></span>
parseQuoted </span>=</span> between (char </span>'"'</span>) (char </span>'"'</span>) (some </span>$</span> anySingleBut </span>'"'</span>)</span></span>
</span>
parseRemember</span> :: Parser Response</span></span>
parseRemember </span>= do</span></span>
  string </span>"remember to "</span></span>
  task </span><-</span> parseQuoted</span></span>
</span>
  remember task</span></span>
</span>
  pure </span>Remember</span></span>
</span>
parseComplete</span> :: Parser Response</span></span>
parseComplete </span>= do</span></span>
  string </span>"mark "</span></span>
  task </span><-</span> parseQuoted</span></span>
  string </span>" as completed"</span></span>
</span>
  forget task</span></span>
</span>
  pure </span>Complete</span></span>
</span>
parseList</span> :: Parser Response</span></span>
parseList </span>= do</span></span>
  string </span>"list tasks"</span></span>
</span>
  tasks </span><-</span> recall</span></span>
</span>
  pure </span>$</span> List</span> tasks</span></span>
</span>
parseRequest</span> :: Parser Response</span></span>
parseRequest </span>=</span> parseRemember </span><|></span> parseComplete </span><|></span> parseList</span></span>
</span>
loop</span> :: Memory -> IO</span> ()</span></span>
loop memory </span>= do</span></span>
  putStr </span>"> "</span></span>
  request </span><-</span> getLine</span></span>
</span>
  let</span> stateAction </span>=</span> runParserT parseRequest </span>"<stdin>"</span> request</span></span>
  let</span> (eitherResponse, newMemory)</span> =</span> runState stateAction memory</span></span>
</span>
  putStrLn </span>$ case</span> eitherResponse </span>of</span></span>
    Left</span> _ </span>-></span> "Sorry, I didn't understand that."</span></span>
    Right</span> response </span>-></span> show response</span></span>
</span>
  loop newMemory</span></span>
</span>
main</span> :: IO</span> ()</span></span>
main </span>= do</span></span>
  hSetBuffering stdout </span>NoBuffering</span></span>
  loop initialMemory</span></span></code></pre>
Pranav - Tutorial

Stateful parsing with Megaparsec
