1 {-# LANGUAGE RankNTypes, TypeFamilies, BangPatterns, Trustworthy #-}
11 -- ** @view@ \/ @(^.)@
14 -- ** @over@ \/ @(%~)@
20 -- * Special types: @Producer Text m (Producer Text m r)@ and @FreeT (Producer Text m) m r@
58 -- * Primitive Character Parsers
94 , module Data.ByteString
96 , module Data.Profunctor
101 import Control.Applicative ((<*))
102 import Control.Monad (liftM, join)
103 import Control.Monad.Trans.State.Strict (StateT(..), modify)
104 import qualified Data.Text as T
105 import Data.Text (Text)
106 import qualified Data.Text.Lazy as TL
107 import Data.ByteString (ByteString)
108 import Data.Functor.Constant (Constant(Constant, getConstant))
109 import Data.Functor.Identity (Identity)
110 import Data.Profunctor (Profunctor)
111 import qualified Data.Profunctor
113 import Pipes.Group (concats, intercalates, FreeT(..), FreeF(..))
114 import qualified Pipes.Group as PG
115 import qualified Pipes.Parse as PP
116 import Pipes.Parse (Parser)
117 import qualified Pipes.Prelude as P
118 import Data.Char (isSpace)
119 import Data.Word (Word8)
120 import Foreign.Storable (sizeOf)
121 import Data.Bits (shiftL)
122 import Prelude hiding (
152 This package provides @pipes@ utilities for /text streams/ or /character streams/,
153 realized as streams of 'Text' chunks. The individual chunks are uniformly /strict/,
154 and thus you will generally want @Data.Text@ in scope. But the type
155 @Producer Text m r@ ,as we are using it, is a sort of /pipes/ equivalent of the lazy @Text@ type.
157 This particular module provides many functions equivalent in one way or another to
158 the pure functions in
159 <https://hackage.haskell.org/package/text-1.1.0.0/docs/Data-Text-Lazy.html Data.Text.Lazy>.
160 They transform, divide, group and fold text streams. Though @Producer Text m r@
161 is the type of \'effectful Text\', the functions in this module are \'pure\'
162 in the sense that they are uniformly monad-independent.
163 Simple /IO/ operations are defined in @Pipes.Text.IO@ -- as lazy IO @Text@
164 operations are in @Data.Text.Lazy.IO@. Inter-operation with @ByteString@
165 is provided in @Pipes.Text.Encoding@, which parallels @Data.Text.Lazy.Encoding@.
167 The Text type exported by @Data.Text.Lazy@ is basically that of a lazy list of
168 strict Text: the implementation is arranged so that the individual strict 'Text'
169 chunks are kept to a reasonable size; the user is not aware of the divisions
170 between the connected 'Text' chunks.
171 So also here: the functions in this module are designed to operate on streams that
172 are insensitive to text boundaries. This means that they may freely split
173 text into smaller texts and /discard empty texts/. The objective, though, is
174 that they should /never concatenate texts/ in order to provide strict upper
175 bounds on memory usage.
177 For example, to stream only the first three lines of 'stdin' to 'stdout' you
181 > import qualified Pipes.Text as Text
182 > import qualified Pipes.Text.IO as Text
183 > import Pipes.Group (takes')
186 > main = runEffect $ takeLines 3 Text.stdin >-> Text.stdout
188 > takeLines n = Text.unlines . takes' n . view Text.lines
190 The above program will never bring more than one chunk of text (~ 32 KB) into
191 memory, no matter how long the lines are.
195 As this example shows, one superficial difference from @Data.Text.Lazy@
196 is that many of the operations, like 'lines', are \'lensified\'; this has a
197 number of advantages (where it is possible); in particular it facilitates their
198 use with 'Parser's of Text (in the general <http://hackage.haskell.org/package/pipes-parse-3.0.1/docs/Pipes-Parse-Tutorial.html pipes-parse>
199 sense.) The disadvantage, famously, is that the messages you get for type errors can be
200 a little alarming. The remarks that follow in this section are for non-lens adepts.
202 Each lens exported here, e.g. 'lines', 'chunksOf' or 'splitAt', reduces to the
203 intuitively corresponding function when used with @view@ or @(^.)@. Instead of
206 > splitAt 17 producer
208 as we would with the Prelude or Text functions, we write
210 > view (splitAt 17) producer
214 > producer ^. splitAt 17
216 This may seem a little indirect, but note that many equivalents of
217 @Text -> Text@ functions are exported here as 'Pipe's. Here too we recover the intuitively
218 corresponding functions by prefixing them with @(>->)@. Thus something like
220 > stripLines = Text.unlines . Group.maps (>-> Text.stripStart) . view Text.lines
222 would drop the leading white space from each line.
224 The lenses in this library are marked as /improper/; this just means that
225 they don't admit all the operations of an ideal lens, but only /getting/ and /focusing/.
226 Just for this reason, though, the magnificent complexities of the lens libraries
227 are a distraction. The lens combinators to keep in mind, the ones that make sense for
228 our lenses, are @view@ \/ @(^.)@), @over@ \/ @(%~)@ , and @zoom@.
230 One need only keep in mind that if @l@ is a @Lens' a b@, then:
234 @view l@ is a function @a -> b@ . Thus @view l a@ (also written @a ^. l@ )
235 is the corresponding @b@; as was said above, this function will be exactly the
236 function you think it is, given its name. Thus to uppercase the first n characters
237 of a Producer, leaving the rest the same, we could write:
240 > upper n p = do p' <- p ^. Text.splitAt n >-> Text.toUpper
244 @over l@ is a function @(b -> b) -> a -> a@. Thus, given a function that modifies
245 @b@s, the lens lets us modify an @a@ by applying @f :: b -> b@ to
246 the @b@ that we can \"see\" through the lens. So @over l f :: a -> a@
247 (it can also be written @l %~ f@).
248 For any particular @a@, then, @over l f a@ or @(l %~ f) a@ is a revised @a@.
249 So above we might have written things like these:
251 > stripLines = Text.lines %~ maps (>-> Text.stripStart)
252 > stripLines = over Text.lines (maps (>-> Text.stripStart))
253 > upper n = Text.splitAt n %~ (>-> Text.toUpper)
257 @zoom l@, finally, is a function from a @Parser b m r@
258 to a @Parser a m r@ (or more generally a @StateT (Producer b m x) m r@).
259 Its use is easiest to see with an decoding lens like 'utf8', which
260 \"sees\" a Text producer hidden inside a ByteString producer:
261 @drawChar@ is a Text parser, returning a @Maybe Char@, @zoom utf8 drawChar@ is
262 a /ByteString/ parser, returning a @Maybe Char@. @drawAll@ is a Parser that returns
263 a list of everything produced from a Producer, leaving only the return value; it would
264 usually be unreasonable to use it. But @zoom (splitAt 17) drawAll@
265 returns a list of Text chunks containing the first seventeen Chars, and returns the rest of
266 the Text Producer for further parsing. Suppose that we want, inexplicably, to
267 modify the casing of a Text Producer according to any instruction it might
268 contain at the start. Then we might write something like this:
270 > obey :: Monad m => Producer Text m b -> Producer Text m b
271 > obey p = do (ts, p') <- lift $ runStateT (zoom (Text.splitAt 7) drawAll) p
272 > let seven = T.concat ts
273 > case T.toUpper seven of
274 > "TOUPPER" -> p' >-> Text.toUpper
275 > "TOLOWER" -> p' >-> Text.toLower
276 > _ -> do yield seven
280 > >>> let doc = each ["toU","pperTh","is document.\n"]
281 > >>> runEffect $ obey doc >-> Text.stdout
284 The purpose of exporting lenses is the mental economy achieved with this three-way
285 applicability. That one expression, e.g. @lines@ or @splitAt 17@ can have these
286 three uses is no more surprising than that a pipe can act as a function modifying
287 the output of a producer, namely by using @>->@ to its left: @producer >-> pipe@
288 -- but can /also/ modify the inputs to a consumer by using @>->@ to its right:
291 The three functions, @view@ \/ @(^.)@, @over@ \/ @(%~)@ and @zoom@ are supplied by
292 both <http://hackage.haskell.org/package/lens lens> and
293 <http://hackage.haskell.org/package/lens-family lens-family> The use of 'zoom' is explained
294 in <http://hackage.haskell.org/package/pipes-parse-3.0.1/docs/Pipes-Parse-Tutorial.html Pipes.Parse.Tutorial>
295 and to some extent in the @Pipes.Text.Encoding@ module here.
299 These simple 'lines' examples reveal a more important difference from @Data.Text.Lazy@ .
300 This is in the types that are most closely associated with our central text type,
301 @Producer Text m r@. In @Data.Text@ and @Data.Text.Lazy@ we find functions like
303 > splitAt :: Int -> Text -> (Text, Text)
304 > lines :: Text -> [Text]
305 > chunksOf :: Int -> Text -> [Text]
307 which relate a Text with a pair of Texts or a list of Texts.
308 The corresponding functions here (taking account of \'lensification\') are
310 > view . splitAt :: (Monad m, Integral n) => n -> Producer Text m r -> Producer Text m (Producer Text m r)
311 > view lines :: Monad m => Producer Text m r -> FreeT (Producer Text m) m r
312 > view . chunksOf :: (Monad m, Integral n) => n -> Producer Text m r -> FreeT (Producer Text m) m r
314 Some of the types may be more readable if you imagine that we have introduced
315 our own type synonyms
317 > type Text m r = Producer T.Text m r
318 > type Texts m r = FreeT (Producer T.Text m) m r
320 Then we would think of the types above as
322 > view . splitAt :: (Monad m, Integral n) => n -> Text m r -> Text m (Text m r)
323 > view lines :: (Monad m) => Text m r -> Texts m r
324 > view . chunksOf :: (Monad m, Integral n) => n -> Text m r -> Texts m r
326 which brings one closer to the types of the similar functions in @Data.Text.Lazy@
328 In the type @Producer Text m (Producer Text m r)@ the second
329 element of the \'pair\' of effectful Texts cannot simply be retrieved
330 with something like 'snd'. This is an \'effectful\' pair, and one must work
331 through the effects of the first element to arrive at the second Text stream, even
332 if you are proposing to throw the Text in the first element away.
333 Note that we use Control.Monad.join to fuse the pair back together, since it specializes to
335 > join :: Monad m => Producer Text m (Producer m r) -> Producer m r
337 The return type of 'lines', 'words', 'chunksOf' and the other /splitter/ functions,
338 @FreeT (Producer m Text) m r@ -- our @Texts m r@ -- is the type of (effectful)
339 lists of (effectful) texts. The type @([Text],r)@ might be seen to gather
340 together things of the forms:
345 > (Text, (Text, (Text, r)))
346 > (Text, (Text, (Text, (Text, r))))
349 (We might also have identified the sum of those types with @Free ((,) Text) r@
350 -- or, more absurdly, @FreeT ((,) Text) Identity r@.)
352 Similarly, our type @Texts m r@, or @FreeT (Text m) m r@ -- in fact called
353 @FreeT (Producer Text m) m r@ here -- encompasses all the members of the sequence:
358 > Text m (Text m (Text m r))
359 > Text m (Text m (Text m (Text m r)))
362 We might have used a more specialized type in place of @FreeT (Producer a m) m r@,
363 or indeed of @FreeT (Producer Text m) m r@, but it is clear that the correct
364 result type of 'lines' will be isomorphic to @FreeT (Producer Text m) m r@ .
368 > lines :: Monad m => Lens' (Producer Text m r) (FreeT (Producer Text m) m r)
369 > view . lines :: Monad m => Producer Text m r -> FreeT (Producer Text m) m r
371 should really have the type
373 > lines :: Monad m => Pipe Text Text m r
375 as e.g. 'toUpper' does. But this would spoil the control we are
376 attempting to maintain over the size of chunks. It is in fact just
377 as unreasonable to want such a pipe as to want
379 > Data.Text.Lazy.lines :: Text -> Text
381 to 'rechunk' the strict Text chunks inside the lazy Text to respect
382 line boundaries. In fact we have
384 > Data.Text.Lazy.lines :: Text -> [Text]
385 > Prelude.lines :: String -> [String]
387 where the elements of the list are themselves lazy Texts or Strings; the use
388 of @FreeT (Producer Text m) m r@ is simply the 'effectful' version of this.
390 The @Pipes.Group@ module, which can generally be imported without qualification,
391 provides many functions for working with things of type @FreeT (Producer a m) m r@.
392 In particular it conveniently exports the constructors for @FreeT@ and the associated
393 @FreeF@ type -- a fancy form of @Either@, namely
395 > data FreeF f a b = Pure a | Free (f b)
397 for pattern-matching. Consider the implementation of the 'words' function, or
398 of the part of the lens that takes us to the words; it is compact but exhibits many
399 of the points under discussion, including explicit handling of the @FreeT@ and @FreeF@
400 constuctors. Keep in mind that
402 > newtype FreeT f m a = FreeT (m (FreeF f a (FreeT f m a)))
403 > next :: Monad m => Producer a m r -> m (Either r (a, Producer a m r))
405 Thus the @do@ block after the @FreeT@ constructor is in the base monad, e.g. 'IO' or 'Identity';
406 the later subordinate block, opened by the @Free@ constructor, is in the @Producer@ monad:
408 > words :: Monad m => Producer Text m r -> FreeT (Producer Text m) m r
409 > words p = FreeT $ do -- With 'next' we will inspect p's first chunk, excluding spaces;
410 > x <- next (p >-> dropWhile isSpace) -- note that 'dropWhile isSpace' is a pipe, and is thus *applied* with '>->'.
411 > return $ case x of -- We use 'return' and so need something of type 'FreeF (Text m) r (Texts m r)'
412 > Left r -> Pure r -- 'Left' means we got no Text chunk, but only the return value; so we are done.
413 > Right (txt, p') -> Free $ do -- If we get a chunk and the rest of the producer, p', we enter the 'Producer' monad
414 > p'' <- view (break isSpace) -- When we apply 'break isSpace', we get a Producer that returns a Producer;
415 > (yield txt >> p') -- so here we yield everything up to the next space, and get the rest back.
416 > return (words p'') -- We then carry on with the rest, which is likely to begin with space.
420 -- | Convert a lazy 'TL.Text' into a 'Producer' of strict 'Text's
421 fromLazy :: (Monad m) => TL.Text -> Producer' Text m ()
422 fromLazy = TL.foldrChunks (\e a -> yield e >> a) (return ())
423 {-# INLINE fromLazy #-}
425 (^.) :: a -> ((b -> Constant b b) -> (a -> Constant b a)) -> b
426 a ^. lens = getConstant (lens Constant a)
428 -- | Apply a transformation to each 'Char' in the stream
429 map :: (Monad m) => (Char -> Char) -> Pipe Text Text m r
430 map f = P.map (T.map f)
431 {-# INLINABLE map #-}
433 -- | Map a function over the characters of a text stream and concatenate the results
435 :: (Monad m) => (Char -> Text) -> Pipe Text Text m r
436 concatMap f = P.map (T.concatMap f)
437 {-# INLINABLE concatMap #-}
439 -- | Transform a Pipe of 'String's into one of 'Text' chunks
440 pack :: Monad m => Pipe String Text m r
442 {-# INLINEABLE pack #-}
444 -- | Transform a Pipes of 'Text' chunks into one of 'String's
445 unpack :: Monad m => Pipe Text String m r
446 unpack = for cat (\t -> yield (T.unpack t))
447 {-# INLINEABLE unpack #-}
449 -- | @toCaseFold@, @toLower@, @toUpper@ and @stripStart@ are standard 'Text' utilities,
450 -- here acting as 'Text' pipes, rather as they would on a lazy text
451 toCaseFold :: Monad m => Pipe Text Text m r
452 toCaseFold = P.map T.toCaseFold
453 {-# INLINEABLE toCaseFold #-}
455 -- | lowercase incoming 'Text'
456 toLower :: Monad m => Pipe Text Text m r
457 toLower = P.map T.toLower
458 {-# INLINEABLE toLower #-}
460 -- | uppercase incoming 'Text'
461 toUpper :: Monad m => Pipe Text Text m r
462 toUpper = P.map T.toUpper
463 {-# INLINEABLE toUpper #-}
465 -- | Remove leading white space from an incoming succession of 'Text's
466 stripStart :: Monad m => Pipe Text Text m r
469 let text = T.stripStart chunk
474 {-# INLINEABLE stripStart #-}
476 -- | @(take n)@ only allows @n@ individual characters to pass;
477 -- contrast @Pipes.Prelude.take@ which would let @n@ chunks pass.
478 take :: (Monad m, Integral a) => a -> Pipe Text Text m ()
479 take n0 = go n0 where
484 let len = fromIntegral (T.length txt)
486 then yield (T.take (fromIntegral n) txt)
490 {-# INLINABLE take #-}
492 -- | @(drop n)@ drops the first @n@ characters
493 drop :: (Monad m, Integral a) => a -> Pipe Text Text m r
494 drop n0 = go n0 where
499 let len = fromIntegral (T.length txt)
502 yield (T.drop (fromIntegral n) txt)
505 {-# INLINABLE drop #-}
507 -- | Take characters until they fail the predicate
508 takeWhile :: (Monad m) => (Char -> Bool) -> Pipe Text Text m ()
509 takeWhile predicate = go
513 let (prefix, suffix) = T.span predicate txt
519 {-# INLINABLE takeWhile #-}
521 -- | Drop characters until they fail the predicate
522 dropWhile :: (Monad m) => (Char -> Bool) -> Pipe Text Text m r
523 dropWhile predicate = go where
526 case T.findIndex (not . predicate) txt of
531 {-# INLINABLE dropWhile #-}
533 -- | Only allows 'Char's to pass if they satisfy the predicate
534 filter :: (Monad m) => (Char -> Bool) -> Pipe Text Text m r
535 filter predicate = P.map (T.filter predicate)
536 {-# INLINABLE filter #-}
538 -- | Strict left scan over the characters
541 => (Char -> Char -> Char) -> Char -> Pipe Text Text m r
543 yield (T.singleton begin)
548 let txt' = T.scanl step c txt
552 {-# INLINABLE scan #-}
554 {-| Fold a pure 'Producer' of strict 'Text's into a lazy
557 toLazy :: Producer Text Identity () -> TL.Text
558 toLazy = TL.fromChunks . P.toList
559 {-# INLINABLE toLazy #-}
561 {-| Fold an effectful 'Producer' of strict 'Text's into a lazy
564 Note: 'toLazyM' is not an idiomatic use of @pipes@, but I provide it for
565 simple testing purposes. Idiomatic @pipes@ style consumes the chunks
566 immediately as they are generated instead of loading them all into memory.
568 toLazyM :: (Monad m) => Producer Text m () -> m TL.Text
569 toLazyM = liftM TL.fromChunks . P.toListM
570 {-# INLINABLE toLazyM #-}
572 -- | Reduce the text stream using a strict left fold over characters
575 => (x -> Char -> x) -> x -> (x -> r) -> Producer Text m () -> m r
576 foldChars step begin done = P.fold (T.foldl' step) begin done
577 {-# INLINABLE foldChars #-}
579 -- | Retrieve the first 'Char'
580 head :: (Monad m) => Producer Text m () -> m (Maybe Char)
586 Left _ -> return Nothing
587 Right (c, _) -> return (Just c)
588 {-# INLINABLE head #-}
590 -- | Retrieve the last 'Char'
591 last :: (Monad m) => Producer Text m () -> m (Maybe Char)
601 else go (Just $ T.last txt) p'
602 {-# INLINABLE last #-}
604 -- | Determine if the stream is empty
605 null :: (Monad m) => Producer Text m () -> m Bool
607 {-# INLINABLE null #-}
609 -- | Count the number of characters in the stream
610 length :: (Monad m, Num n) => Producer Text m () -> m n
611 length = P.fold (\n txt -> n + fromIntegral (T.length txt)) 0 id
612 {-# INLINABLE length #-}
614 -- | Fold that returns whether 'M.Any' received 'Char's satisfy the predicate
615 any :: (Monad m) => (Char -> Bool) -> Producer Text m () -> m Bool
616 any predicate = P.any (T.any predicate)
617 {-# INLINABLE any #-}
619 -- | Fold that returns whether 'M.All' received 'Char's satisfy the predicate
620 all :: (Monad m) => (Char -> Bool) -> Producer Text m () -> m Bool
621 all predicate = P.all (T.all predicate)
622 {-# INLINABLE all #-}
624 -- | Return the maximum 'Char' within a text stream
625 maximum :: (Monad m) => Producer Text m () -> m (Maybe Char)
626 maximum = P.fold step Nothing id
631 else Just $ case mc of
632 Nothing -> T.maximum txt
633 Just c -> max c (T.maximum txt)
634 {-# INLINABLE maximum #-}
636 -- | Return the minimum 'Char' within a text stream (surely very useful!)
637 minimum :: (Monad m) => Producer Text m () -> m (Maybe Char)
638 minimum = P.fold step Nothing id
644 Nothing -> Just (T.minimum txt)
645 Just c -> Just (min c (T.minimum txt))
646 {-# INLINABLE minimum #-}
648 -- | Find the first element in the stream that matches the predicate
651 => (Char -> Bool) -> Producer Text m () -> m (Maybe Char)
652 find predicate p = head (p >-> filter predicate)
653 {-# INLINABLE find #-}
655 -- | Index into a text stream
657 :: (Monad m, Integral a)
658 => a-> Producer Text m () -> m (Maybe Char)
659 index n p = head (p >-> drop n)
660 {-# INLINABLE index #-}
663 -- | Store a tally of how many segments match the given 'Text'
664 count :: (Monad m, Num n) => Text -> Producer Text m () -> m n
665 count c p = P.fold (+) 0 id (p >-> P.map (fromIntegral . T.count c))
666 {-# INLINABLE count #-}
669 -- | Consume the first character from a stream of 'Text'
671 -- 'next' either fails with a 'Left' if the 'Producer' has no more characters or
672 -- succeeds with a 'Right' providing the next character and the remainder of the
678 -> m (Either r (Char, Producer Text m r))
684 Left r -> return (Left r)
685 Right (txt, p') -> case (T.uncons txt) of
687 Just (c, txt') -> return (Right (c, yield txt' >> p'))
688 {-# INLINABLE nextChar #-}
690 -- | Draw one 'Char' from a stream of 'Text', returning 'Left' if the 'Producer' is empty
692 drawChar :: (Monad m) => Parser Text m (Maybe Char)
696 Nothing -> return Nothing
697 Just txt -> case (T.uncons txt) of
702 {-# INLINABLE drawChar #-}
704 -- | Push back a 'Char' onto the underlying 'Producer'
705 unDrawChar :: (Monad m) => Char -> Parser Text m ()
706 unDrawChar c = modify (yield (T.singleton c) >>)
707 {-# INLINABLE unDrawChar #-}
709 {-| 'peekChar' checks the first 'Char' in the stream, but uses 'unDrawChar' to
715 > Left _ -> return ()
716 > Right c -> unDrawChar c
721 peekChar :: (Monad m) => Parser Text m (Maybe Char)
726 Just c -> unDrawChar c
728 {-# INLINABLE peekChar #-}
730 {-| Check if the underlying 'Producer' has no more characters
732 Note that this will skip over empty 'Text' chunks, unlike
733 'PP.isEndOfInput' from @pipes-parse@, which would consider
734 an empty 'Text' a valid bit of input.
736 > isEndOfChars = liftM isLeft peekChar
738 isEndOfChars :: (Monad m) => Parser Text m Bool
744 {-# INLINABLE isEndOfChars #-}
747 -- | Splits a 'Producer' after the given number of characters
749 :: (Monad m, Integral n)
751 -> Lens' (Producer Text m r)
752 (Producer Text m (Producer Text m r))
753 splitAt n0 k p0 = fmap join (k (go n0 p0))
759 Left r -> return (return r)
760 Right (txt, p') -> do
761 let len = fromIntegral (T.length txt)
767 let (prefix, suffix) = T.splitAt (fromIntegral n) txt
769 return (yield suffix >> p')
770 {-# INLINABLE splitAt #-}
773 -- | Split a text stream in two, producing the longest
774 -- consecutive group of characters that satisfies the predicate
775 -- and returning the rest
780 -> Lens' (Producer Text m r)
781 (Producer Text m (Producer Text m r))
782 span predicate k p0 = fmap join (k (go p0))
787 Left r -> return (return r)
788 Right (txt, p') -> do
789 let (prefix, suffix) = T.span predicate txt
796 return (yield suffix >> p')
797 {-# INLINABLE span #-}
799 {-| Split a text stream in two, producing the longest
800 consecutive group of characters that don't satisfy the predicate
805 -> Lens' (Producer Text m r)
806 (Producer Text m (Producer Text m r))
807 break predicate = span (not . predicate)
808 {-# INLINABLE break #-}
810 {-| Improper lens that splits after the first group of equivalent Chars, as
811 defined by the given equivalence relation
815 => (Char -> Char -> Bool)
816 -> Lens' (Producer Text m r)
817 (Producer Text m (Producer Text m r))
818 groupBy equals k p0 = fmap join (k ((go p0))) where
822 Left r -> return (return r)
823 Right (txt, p') -> case T.uncons txt of
825 Just (c, _) -> (yield txt >> p') ^. span (equals c)
826 {-# INLINABLE groupBy #-}
828 -- | Improper lens that splits after the first succession of identical 'Char' s
830 => Lens' (Producer Text m r)
831 (Producer Text m (Producer Text m r))
833 {-# INLINABLE group #-}
835 {-| Improper lens that splits a 'Producer' after the first word
837 Unlike 'words', this does not drop leading whitespace
840 => Lens' (Producer Text m r)
841 (Producer Text m (Producer Text m r))
842 word k p0 = fmap join (k (to p0))
845 p' <- p^.span isSpace
847 {-# INLINABLE word #-}
851 => Lens' (Producer Text m r)
852 (Producer Text m (Producer Text m r))
853 line = break (== '\n')
855 {-# INLINABLE line #-}
858 -- | Intersperse a 'Char' in between the characters of stream of 'Text'
860 :: (Monad m) => Char -> Producer Text m r -> Producer Text m r
867 Right (txt, p') -> do
868 yield (T.intersperse c txt)
874 Right (txt, p') -> do
875 yield (T.singleton c)
876 yield (T.intersperse c txt)
878 {-# INLINABLE intersperse #-}
881 -- | Improper isomorphism between a 'Producer' of 'ByteString's and 'Word8's
882 packChars :: Monad m => Iso'_ (Producer Char m x) (Producer Text m x)
883 packChars = Data.Profunctor.dimap to (fmap from)
885 -- to :: Monad m => Producer Char m x -> Producer Text m x
886 to p = PG.folds step id done (p^.PG.chunksOf defaultChunkSize)
888 step diffAs c = diffAs . (c:)
890 done diffAs = T.pack (diffAs [])
892 -- from :: Monad m => Producer Text m x -> Producer Char m x
893 from p = for p (each . T.unpack)
894 {-# INLINABLE packChars #-}
896 defaultChunkSize :: Int
897 defaultChunkSize = 16384 - (sizeOf (undefined :: Int) `shiftL` 1)
899 -- | Split a text stream into 'FreeT'-delimited text streams of fixed size
901 :: (Monad m, Integral n)
902 => n -> Lens' (Producer Text m r)
903 (FreeT (Producer Text m) m r)
904 chunksOf n k p0 = fmap concats (k (FreeT (go p0)))
910 Right (txt, p') -> Free $ do
911 p'' <- (yield txt >> p') ^. splitAt n
912 return $ FreeT (go p'')
913 {-# INLINABLE chunksOf #-}
916 {-| Split a text stream into sub-streams delimited by characters that satisfy the
923 -> FreeT (Producer Text m) m r
924 splitsWith predicate p0 = FreeT (go0 p0)
929 Left r -> return (Pure r)
933 else return $ Free $ do
934 p'' <- (yield txt >> p') ^. span (not . predicate)
935 return $ FreeT (go1 p'')
940 Right (_, p') -> Free $ do
941 p'' <- p' ^. span (not . predicate)
942 return $ FreeT (go1 p'')
943 {-# INLINABLE splitsWith #-}
945 -- | Split a text stream using the given 'Char' as the delimiter
948 -> Lens' (Producer Text m r)
949 (FreeT (Producer Text m) m r)
951 fmap (PG.intercalates (yield (T.singleton c))) (k (splitsWith (c ==) p))
952 {-# INLINABLE splits #-}
954 {-| Isomorphism between a stream of 'Text' and groups of equivalent 'Char's , using the
955 given equivalence relation
959 => (Char -> Char -> Bool)
960 -> Lens' (Producer Text m x) (FreeT (Producer Text m) m x)
961 groupsBy equals k p0 = fmap concats (k (FreeT (go p0))) where
962 go p = do x <- next p
963 case x of Left r -> return (Pure r)
964 Right (bs, p') -> case T.uncons bs of
966 Just (c, _) -> do return $ Free $ do
967 p'' <- (yield bs >> p')^.span (equals c)
968 return $ FreeT (go p'')
969 {-# INLINABLE groupsBy #-}
972 -- | Like 'groupsBy', where the equality predicate is ('==')
975 => Lens' (Producer Text m x) (FreeT (Producer Text m) m x)
976 groups = groupsBy (==)
977 {-# INLINABLE groups #-}
981 {-| Split a text stream into 'FreeT'-delimited lines
984 :: (Monad m) => Iso'_ (Producer Text m r) (FreeT (Producer Text m) m r)
985 lines = Data.Profunctor.dimap _lines (fmap _unlines)
987 _lines p0 = FreeT (go0 p0)
992 Left r -> return (Pure r)
996 else return $ Free $ go1 (yield txt >> p')
998 p' <- p ^. break ('\n' ==)
1002 Left r -> return $ Pure r
1003 Right (_, p'') -> go0 p''
1006 -- => FreeT (Producer Text m) m x -> Producer Text m x
1007 _unlines = concats . PG.maps (<* yield (T.singleton '\n'))
1009 {-# INLINABLE lines #-}
1012 -- | Split a text stream into 'FreeT'-delimited words
1014 :: (Monad m) => Iso'_ (Producer Text m r) (FreeT (Producer Text m) m r)
1015 words = Data.Profunctor.dimap go (fmap _unwords)
1018 x <- next (p >-> dropWhile isSpace)
1021 Right (bs, p') -> Free $ do
1022 p'' <- (yield bs >> p') ^. break isSpace
1024 _unwords = PG.intercalates (yield $ T.singleton ' ')
1026 {-# INLINABLE words #-}
1029 {-| 'intercalate' concatenates the 'FreeT'-delimited text streams after
1030 interspersing a text stream in between them
1034 => Producer Text m ()
1035 -> FreeT (Producer Text m) m r
1036 -> Producer Text m r
1037 intercalate p0 = go0
1040 x <- lift (runFreeT f)
1047 x <- lift (runFreeT f)
1054 {-# INLINABLE intercalate #-}
1056 {-| Join 'FreeT'-delimited lines into a text stream
1059 :: (Monad m) => FreeT (Producer Text m) m r -> Producer Text m r
1063 x <- lift (runFreeT f)
1068 yield $ T.singleton '\n'
1070 {-# INLINABLE unlines #-}
1072 {-| Join 'FreeT'-delimited words into a text stream
1075 :: (Monad m) => FreeT (Producer Text m) m r -> Producer Text m r
1076 unwords = intercalate (yield $ T.singleton ' ')
1077 {-# INLINABLE unwords #-}
1082 @Data.Text@ re-exports the 'Text' type.
1084 @Pipes.Parse@ re-exports 'input', 'concat', 'FreeT' (the type) and the 'Parse' synonym.
1088 type Lens' a b = forall f . Functor f => (b -> f b) -> (a -> f a)