aboutsummaryrefslogtreecommitdiffhomepage
path: root/Pipes
diff options
context:
space:
mode:
Diffstat (limited to 'Pipes')
-rw-r--r--Pipes/Prelude/Text.hs77
-rw-r--r--Pipes/Text/Encoding.hs15
2 files changed, 52 insertions, 40 deletions
diff --git a/Pipes/Prelude/Text.hs b/Pipes/Prelude/Text.hs
index 701d5f6..46424a7 100644
--- a/Pipes/Prelude/Text.hs
+++ b/Pipes/Prelude/Text.hs
@@ -31,14 +31,18 @@ import Prelude hiding (readFile, writeFile)
31 Line-based operations are marked with a final \-@Ln@, like 'stdinLn', 'readFileLn', etc. 31 Line-based operations are marked with a final \-@Ln@, like 'stdinLn', 'readFileLn', etc.
32 They are drop-in 'Text' replacements for the corresponding 'String' operations in 32 They are drop-in 'Text' replacements for the corresponding 'String' operations in
33 @Pipes.Prelude@ and @Pipes.Safe.Prelude@ - a final \-@Ln@ being added where necessary. 33 @Pipes.Prelude@ and @Pipes.Safe.Prelude@ - a final \-@Ln@ being added where necessary.
34 In using them, one is producing and consuming semantically significant individual texts, 34 This module can thus be imported unqualified if @Pipes.Prelude@ is imported qualified, as
35 it must be.
36
37 In using the line-based operations, one is producing and consuming semantically significant individual texts,
35 understood as lines, just as one would produce or pipe 'Int's or 'Char's or anything else. 38 understood as lines, just as one would produce or pipe 'Int's or 'Char's or anything else.
36 The standard materials from @Pipes@ and @Pipes.Prelude@ and 39 The standard materials from @Pipes@ and @Pipes.Prelude@ and
37 @Data.Text@ are all you need to work with them, and 40 @Data.Text@ are all you need to work with them, and
38 you can use these operations without using any of the other modules in this package. 41 you can use these operations without using any of the other modules in this package.
39 42
40 Thus, to take a trivial case, here we upper-case three lines from standard input and write 43 Thus, to take a trivial case, here we upper-case three lines from standard input and write
41 them to a file. 44 them to a file. (@runSafeT@ from @Pipes.Safe@ just makes sure to close any handles opened in its scope;
45 it is only needed for @readFileLn@ and @writeFileLn@.)
42 46
43>>> import Pipes 47>>> import Pipes
44>>> import qualified Pipes.Prelude as P 48>>> import qualified Pipes.Prelude as P
@@ -53,8 +57,7 @@ ONE
53TWO 57TWO
54THREE 58THREE
55 59
56 Here @runSafeT@ from @Pipes.Safe@ just makes sure to close any handles opened in its scope. 60 The point of view is very much that of @Pipes.Prelude@, substituting @Text@ for @String@.
57 Otherwise the point of view is very much that of @Pipes.Prelude@, substituting @Text@ for @String@.
58 It would still be the same even if 61 It would still be the same even if
59 we did something a bit more sophisticated, like run an ordinary attoparsec 'Text' parser on 62 we did something a bit more sophisticated, like run an ordinary attoparsec 'Text' parser on
60 each line, as is frequently desirable. Here we use 63 each line, as is frequently desirable. Here we use
@@ -71,37 +74,40 @@ quit<Enter>
71[1.0,2.0,3.0] 74[1.0,2.0,3.0]
72 75
73 The line-based operations are, however, subject to a number of caveats. 76 The line-based operations are, however, subject to a number of caveats.
74 First, where they read from a handle, they will of course happily 77
75 accumulate indefinitely long lines. This is likely to be legitimate for input 78 * Where these line-based operations read from a handle, they will
76 typed in by a user, and for locally produced files of known characteristics, but 79 accumulate indefinitely long lines. This makes sense for input
77 otherwise not. See the post on 80 typed in by a user, and for locally produced files of known characteristics, but
78 <http://www.haskellforall.com/2013/09/perfect-streaming-using-pipes-bytestring.html perfect streaming> 81 otherwise not. See the post on
79 to see why @pipes-bytestring@ and this package, outside this module, take a different approach. 82 <http://www.haskellforall.com/2013/09/perfect-streaming-using-pipes-bytestring.html perfect streaming>
80 Furthermore, the line-based operations, 83 to see why @pipes-bytestring@ and this package, outside this module, take a different approach, in which
81 like those in @Data.Text.IO@, use the system encoding (and @T.hGetLine@, @T.hPutLine@ etc.) 84 lines themselves are permitted to stream without accumulation.
82 and thus are slower than the \'official\' route, which would use the very fast 85
83 bytestring IO operations from @Pipes.ByteString@ and 86 * The line-based operations,
84 encoding and decoding functions in @Pipes.Text.Encoding@. Finally, the line-based 87 like those in @Data.Text.IO@, use the system encoding (and @T.hGetLine@, @T.hPutLine@ etc.)
85 operations will generate text exceptions after the fashion of 88 and thus are slower than the \'official\' route, which would use the very fast
86 @Data.Text.Encoding@, rather than returning the undigested bytes in the 89 bytestring IO operations from @Pipes.ByteString@ and the
87 style of @Pipes.Text.Encoding@. 90 encoding and decoding functions in @Pipes.Text.Encoding@, which are also quite fast
91 thanks to the @streaming-commons@ package.
92
93 * The line-based operations (again like those in @Data.Text.IO@) will
94 generate text exceptions after the fashion of
95 @Data.Text.Encoding@, rather than returning the undigested bytes in the
96 style of @Pipes.Text.Encoding@. This is the standard practice in the pipes libraries.
88 97
89-} 98-}
90 99
91 100
92{-| Read separate lines of 'Text' from 'IO.stdin' using 'T.getLine' 101{-| Read separate lines of 'Text' from 'IO.stdin' using 'T.getLine', terminating on end of input.
93 This function will accumulate indefinitely long strict 'Text's. See the caveats above.
94 102
95 Terminates on end of input 103 This function will accumulate indefinitely long strict 'Text's. See the caveats above.
96-} 104-}
97stdinLn :: MonadIO m => Producer' T.Text m () 105stdinLn :: MonadIO m => Producer' T.Text m ()
98stdinLn = fromHandleLn IO.stdin 106stdinLn = fromHandleLn IO.stdin
99{-# INLINABLE stdinLn #-} 107{-# INLINABLE stdinLn #-}
100 108
101 109
102{-| Write 'Text' lines to 'IO.stdout' using 'putStrLn' 110{-| Write 'Text' lines to 'IO.stdout' using 'putStrLn', terminating without error on a broken output pipe
103
104 Unlike 'toHandle', 'stdoutLn' gracefully terminates on a broken output pipe
105-} 111-}
106stdoutLn :: MonadIO m => Consumer' T.Text m () 112stdoutLn :: MonadIO m => Consumer' T.Text m ()
107stdoutLn = go 113stdoutLn = go
@@ -118,10 +124,8 @@ stdoutLn = go
118 Right () -> go 124 Right () -> go
119{-# INLINABLE stdoutLn #-} 125{-# INLINABLE stdoutLn #-}
120 126
121{-| Write lines of 'Text' to 'IO.stdout'. 127{-| Write lines of 'Text' to 'IO.stdout'. This does not handle a broken output pipe,
122 128 but has a polymorphic return value.
123 This does not handle a broken output pipe, but has a polymorphic return
124 value.
125-} 129-}
126stdoutLn' :: MonadIO m => Consumer' T.Text m r 130stdoutLn' :: MonadIO m => Consumer' T.Text m r
127stdoutLn' = for cat (\str -> liftIO (T.putStrLn str)) 131stdoutLn' = for cat (\str -> liftIO (T.putStrLn str))
@@ -132,10 +136,10 @@ stdoutLn' = for cat (\str -> liftIO (T.putStrLn str))
132 p >-> stdoutLn' = for p (\str -> liftIO (T.putStrLn str)) 136 p >-> stdoutLn' = for p (\str -> liftIO (T.putStrLn str))
133 #-} 137 #-}
134 138
135{-| Read separate lines of 'Text' from a 'IO.Handle' using 'T.hGetLine'. 139{-| Read separate lines of 'Text' from a 'IO.Handle' using 'T.hGetLine',
136 This operation will accumulate indefinitely large strict texts. See the caveats above. 140 terminating at the end of input
137 141
138 Terminates on end of input 142 This operation will accumulate indefinitely large strict texts. See the caveats above.
139-} 143-}
140fromHandleLn :: MonadIO m => IO.Handle -> Producer' Text m () 144fromHandleLn :: MonadIO m => IO.Handle -> Producer' Text m ()
141fromHandleLn h = go where 145fromHandleLn h = go where
@@ -173,8 +177,11 @@ toHandleLn handle = for cat (\str -> liftIO (T.hPutStrLn handle str))
173 #-} 177 #-}
174 178
175 179
176{-| Stream separate lines of text from a file. This operation will accumulate 180{-| Stream separate lines of text from a file. Apply @runSafeT@ after running the
177 indefinitely long strict text chunks. See the caveats above. 181 pipeline to manage the opening and closing of the handle.
182
183 This operation will accumulate indefinitely long strict text chunks.
184 See the caveats above.
178-} 185-}
179readFileLn :: MonadSafe m => FilePath -> Producer Text m () 186readFileLn :: MonadSafe m => FilePath -> Producer Text m ()
180readFileLn file = Safe.withFile file IO.ReadMode fromHandleLn 187readFileLn file = Safe.withFile file IO.ReadMode fromHandleLn
@@ -182,8 +189,8 @@ readFileLn file = Safe.withFile file IO.ReadMode fromHandleLn
182 189
183 190
184 191
185{-| Write lines to a file, automatically opening and closing the file as 192{-| Write lines to a file. Apply @runSafeT@ after running the
186 necessary 193 pipeline to manage the opening and closing of the handle.
187-} 194-}
188writeFileLn :: (MonadSafe m) => FilePath -> Consumer' Text m r 195writeFileLn :: (MonadSafe m) => FilePath -> Consumer' Text m r
189writeFileLn file = Safe.withFile file IO.WriteMode toHandleLn 196writeFileLn file = Safe.withFile file IO.WriteMode toHandleLn
diff --git a/Pipes/Text/Encoding.hs b/Pipes/Text/Encoding.hs
index e242411..a576a42 100644
--- a/Pipes/Text/Encoding.hs
+++ b/Pipes/Text/Encoding.hs
@@ -220,14 +220,17 @@ type Codec
220 220
221 221
222{- | @decode@ is just the ordinary @view@ or @(^.)@ of the lens libraries; 222{- | @decode@ is just the ordinary @view@ or @(^.)@ of the lens libraries;
223 exported here under a name appropriate to the material. Thus 223 exported here under a name appropriate to the material.
224 Thus given a bytestring producer called @bytes@ we have
224 225
225> decode utf8 bytes :: Producer Text IO (Producer ByteString IO ()) 226> decode utf8 bytes :: Producer Text IO (Producer ByteString IO ())
226 227
227 All of these are thus the same: 228 All of these are thus the same:
228 229
229> decode utf8 bytes = view utf8 bytes = bytes ^. utf8 = decodeUtf8 bytes 230> decode utf8 bytes
230 231> view utf8 bytes
232> bytes ^. utf8
233> decodeUtf8 bytes
231 234
232-} 235-}
233 236
@@ -244,9 +247,11 @@ decode codec a = getConstant (codec Constant a)
244 returned as a Left value; in the happy case, a Right value is returned 247 returned as a Left value; in the happy case, a Right value is returned
245 with the anticipated return value for the original bytestring producer. 248 with the anticipated return value for the original bytestring producer.
246 249
247 Again, all of these are the same 250 Given a bytestring producer called @bytes@ all of these will be the same:
248 251
249> decode (utf8 . eof) bytes = view (utf8 . eof) p = p^.utf8.eof 252> decode (utf8 . eof) bytes
253> view (utf8 . eof) bytes
254> bytes^.utf8.eof
250 255
251-} 256-}
252 257