umgeher/jsx
0
Fork 0
Code Issues Pull requests Activity

Merge branch 'develop'

Browse source
This commit is contained in:
alisdair sullivan 2012-04-22 21:28:28 -07:00
commit 0acec78044
19 changed files with 1312 additions and 860 deletions
Download patch file Download diff file Expand all files Collapse all files

57
README.markdown
View file

@ -6,7 +6,7 @@ copyright 2011, 2012 alisdair sullivan
jsx is released under the terms of the [MIT][MIT] license
jsx uses [rebar][rebar] for it's build chain and [meck][meck] for it's test suite
jsx uses [rebar][rebar] for it's build chain
[![Build Status](https://secure.travis-ci.org/talentdeficit/jsx.png?branch=master)](http://travis-ci.org/talentdeficit/jsx)
@ -60,7 +60,7 @@ to minify a json string: `jsx:format(JSON)`
json must be a binary encoded in `utf8`. if it's invalid `utf8` or invalid json, it probably won't parse without errors. there are a few non-standard extensions to the parser available that may change that, they are detailed in the options section below
jsx also supports json fragments; valid json values that are not complete json. that means jsx will parse things like `<<"1">`, `<<"true">>` and `<<"\"hello world\"">>` without problems
jsx also supports json fragments; valid json values that are not complete json. that means jsx will parse things like `<<"1">>`, `<<"true">>` and `<<"\"hello world\"">>` without complaint
#### erlang ####
@ -74,7 +74,7 @@ when converting from erlang to json, numbers are represented with their shortest
#### strings ####
the [json spec][rfc4627] is frustratingly vague on the exact details of json strings. json must be unicode, but no encoding is specified. javascript explicitly allows strings containing codepoints explicitly disallowed by unicode. json allows implementations to set limits on the content of strings and other implementations attempt to resolve this in various ways. this implementation, in default operation, only accepts strings that meet the constraints set out in the json spec (properly escaped control characters, `"` and the escape character, `\`) and that are encoded in `utf8`
the [json spec][rfc4627] is frustratingly vague on the exact details of json strings. json must be unicode, but no encoding is specified. javascript explicitly allows strings containing codepoints explicitly disallowed by unicode. json allows implementations to set limits on the content of strings and other implementations attempt to resolve this in various ways. this implementation, in default operation, only accepts strings that meet the constraints set out in the json spec (strings are sequences of unicode codepoints deliminated by `"` (`u+0022`) that may not contain control codes unless properly escaped with `\` (`u+005c`)) and that are encoded in `utf8`
the utf8 restriction means improperly paired surrogates are explicitly disallowed. `u+d800` to `u+dfff` are allowed, but only when they form valid surrogate pairs. surrogates that appear otherwise are an error
@ -82,7 +82,7 @@ json string escapes of the form `\uXXXX` will be converted to their equivalent c
in the interests of pragmatism, there is an option for looser parsing, see options below
all erlang strings are represented by *valid* `utf8` encoded binaries. the encoder will check strings for conformance. the same restrictions apply as for strings encountered within json texts. that means no unpaired surrogates
all erlang strings are represented by *valid* `utf8` encoded binaries. the encoder will check strings for conformance. noncharacters (like `u+ffff`) are allowed in erlang utf8 encoded binaries, but not in strings passed to the encoder (although see options below)
this implementation performs no normalization on strings beyond that detailed here. be careful when comparing strings as equivalent strings may have different `utf8` encodings
@ -101,47 +101,57 @@ json objects are represented by erlang proplists. the empty object has the speci
### <a name="options">options</a> ###
jsx functions all take a common set of options. not all flags have meaning in all contexts, but they are always valid options. flags are always atoms and have no value. functions may have additional options beyond these, see individual function documentation for details
jsx functions all take a common set of options. not all flags have meaning in all contexts, but they are always valid options. flags are always atoms or {atom, Term} tuples. functions may have additional options beyond these, see individual function documentation for details
#### `loose_unicode` ####
#### `replaced_bad_utf8` ####
json text input and json strings SHOULD be utf8 encoded binaries, appropriately escaped as per the json spec. if this option is present attempts are made to replace invalid codepoints with `u+FFFD` as per the unicode spec. this applies both to malformed unicode and disallowed codepoints
#### `escape_forward_slash` ####
#### `escaped_forward_slashes` ####
json strings are escaped according to the json spec. this means forward slashes are never escaped. unfortunately, a microsoft implementation of json uses escaped forward slashes in json formatted date strings. without this option it is impossible to get date strings that some microsoft tools understand
#### `explicit_end` ####
#### `single_quoted_strings` ####
this option treats all exhausted inputs as incomplete, as explained below. the parser will not attempt to return a final state until the function is called with the value `end_stream`
some parsers allow double quotes (`u+0022`) to be replaced by single quotes (`u+0027`) to deliminate keys and strings. this option allows json containing single quotes as structural (deliminator) characters to be parsed without errors. note that the parser expects strings to be terminated by the same quote type that opened it and that single quotes must, obviously, be escaped within strings deliminated by single quotes
#### `single_quotes` ####
double quotes must ALWAYS be escaped, regardless of what kind of quotes deliminate the string they are found in
some parsers allow double quotes (`u+0022`) to be replaced by single quotes (`u+0027`) to deliminate keys and strings. this option allows json containing single quotes as structural (deliminator) characters to be parsed without errors. note that the parser expects strings to be terminated by the same quote type that opened it and that single quotes must, obviously, be escaped within strings deliminated by single quotes. the parser will never emit json with keys or strings deliminated by single quotes
the parser will never emit json with keys or strings deliminated by single quotes
#### `no_jsonp_escapes` ####
#### `unescaped_jsonp` ####
javascript interpreters treat the codepoints `u+2028` and `u+2029` as significant whitespace. json strings that contain either of these codepoints will be parsed incorrectly by some javascript interpreters. by default, these codepoints are escaped (to `"\u2028"` and `\u2029`, respectively) to retain compatibility. this option simply removes that escaping if, for some reason, you object to this
javascript interpreters treat the codepoints `u+2028` and `u+2029` as significant whitespace. json strings that contain either of these codepoints will be parsed incorrectly by some javascript interpreters. by default, these codepoints are escaped (to `\u2028` and `\u2029`, respectively) to retain compatibility. this option simply removes that escaping if, for some reason, you object to this
#### `comments` ####
json has no official comments but some parsers allow c style comments. this flag allows comments (both `// ...` and `/* ... */` style) anywhere whitespace is allowed
#### `json_escape` ####
#### `escaped_strings` ####
by default, both the encoder and decoder return strings as utf8 binaries appropriate for use in erlang. escape sequences that were present in decoded terms are converted into the appropriate codepoint and encoded terms are unaltered. this flag escapes strings for output in json, removing control codes and replacing them with the appropriate escapes
by default, both the encoder and decoder return strings as utf8 binaries appropriate for use in erlang. escape sequences that were present in decoded terms are converted into the appropriate codepoint and encoded terms are unaltered. this flag escapes strings as if for output in json, removing control codes and problematic codepoints and replacing them with the appropriate escapes
#### `dirty_strings` ####
json escaping is lossy, it mutates the json string and repeated application can result in unwanted behaviour. if your strings are already escaped (or you'd like to force invalid strings into "json") use this flag to bypass escaping
#### `ignore_bad_escapes` ####
#### `ignored_bad_escapes` ####
during decoding, ignore unrecognized escape sequences and leave them as is in the stream
during decoding, ignore unrecognized escape sequences and leave them as is in the stream. note that if you combine this option with `escaped_strings` the escape character itself will be escaped
#### `explicit_end` ####
this option treats all exhausted inputs as incomplete, as explained below. the parser will not attempt to return a final state until the function is called with the value `end_stream`
#### `relax` ####
relax is a synonym for `[loose_unicode, single_quotes, comments, ignore_bad_escapes]`
relax is a synonym for `[replaced_bad_utf8, single_quoted_strings, comments, ignored_bad_escapes]` for when you don't care how janky and awful your json input is, you just want the parser to do the best it can
#### `{pre_encode, F}` ####
`F` is a function of arity 1 that pre-process input to the encoder. only input evaluated in a *value* context is pre-processed in this manner (so keys are not pre-processed, but objects and arrays are). if more than one pre encoder is declared, a `badarg` exception will occur
input can be any term, but output from the function must be a valid type for input
### <a name="incompletes">incomplete input</a> ###
@ -229,9 +239,20 @@ types:
* `binary`
* `atom`
* `existing_atom`
- `{post_decode, F}`
the option `labels` controls how keys are converted from json to erlang terms. `binary` does no conversion beyond normal escaping. `atom` converts keys to erlang atoms, and results in a badarg error if keys fall outside the range of erlang atoms. `existing_atom` is identical to `atom`, except it will not add new atoms to the atom table
`{post_decode, F}` is a user defined function of arity 1 that is called on each output value (objects, arrays, strings, numbers and literals). it may return any value to be substituted in the returned term. for example:
```erlang
1> F = fun(V) when is_list(V) -> V; (V) -> false end.
2> jsx:to_term(<<"{\"a list\": [true, \"a string\", 1]}">>, [{post_decode, F}]).
[{<<"a list">>, [false, false, false]}]
```
if more than one decoder is declared a badarg exception will result
#### converting erlang terms to json ####
`to_json` parses an erlang term and produces a JSON text (see json <-> erlang mapping details below)

185
bin/jsx_bench.escript Executable file
View file

@ -0,0 +1,185 @@
#!/usr/bin/env escript
%% The MIT License
%% Copyright (c) 2012 Alisdair Sullivan <alisdairsullivan@yahoo.ca>
%% Permission is hereby granted, free of charge, to any person obtaining a copy
%% of this software and associated documentation files (the "Software"), to deal
%% in the Software without restriction, including without limitation the rights
%% to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
%% copies of the Software, and to permit persons to whom the Software is
%% furnished to do so, subject to the following conditions:
%% The above copyright notice and this permission notice shall be included in
%% all copies or substantial portions of the Software.
%% THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
%% IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
%% FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
%% AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
%% LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
%% OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
%% THE SOFTWARE.
-mode(compile).
-define(averageN(Test, N),
{average, {repeat, N, Test}}
).
main([]) ->
%% preload jsx mods
jsx:to_term(<<"{}">>),
format(frequency:profile({"empty object to term", ?averageN({jsx, to_term, [<<"{}">>]}, 1000)})),
format(frequency:profile({"empty object to json", ?averageN({jsx, to_json, [[{}]]}, 1000)})),
format(frequency:profile({"empty array to term", ?averageN({jsx, to_term, [<<"[]">>]}, 1000)})),
format(frequency:profile({"empty array to json", ?averageN({jsx, to_json, [[]]}, 1000)})),
format(frequency:profile({"sample tweet to term", ?averageN({jsx, to_term, [sample_tweet()]}, 1000)})),
format(frequency:profile({"sample tweet to json", ?averageN({jsx, to_json, [jsx:to_term(sample_tweet())]}, 1000)})),
format(frequency:profile({"sample github user to term", ?averageN({jsx, to_term, [sample_github_user()]}, 1000)})),
format(frequency:profile({"sample github user to json", ?averageN({jsx, to_json, [jsx:to_term(sample_github_user())]}, 1000)})).
format([]) -> ok;
format([{name, Name}|Rest]) ->
io:format("name : ~p~n", [Name]),
format(Rest);
format([{time, Time}|Rest]) ->
io:format("time : ~p~n", [Time]),
format(Rest);
format([{error, Error}|Rest]) ->
io:format("error : ~p~n", [Error]),
format(Rest);
format([Result|Rest]) when is_list(Result) -> format(Result), format(Rest);
format([_|Rest]) -> format(Rest).
sample_tweet() ->
<<"{
\"coordinates\": null,
\"created_at\": \"Sat Sep 10 22:23:38 +0000 2011\",
\"truncated\": false,
\"favorited\": false,
\"id_str\": \"112652479837110273\",
\"entities\": {
\"urls\": [{
\"expanded_url\": \"http://instagr.am/p/MuW67/\",
\"url\": \"http://t.co/6J2EgYM\",
\"indices\": [67, 86],
\"display_url\": \"instagr.am/p/MuW67/\"
}],
\"hashtags\": [{
\"text\": \"tcdisrupt\",
\"indices\": [32,42]
}],
\"user_mentions\": [
{
\"name\": \"Twitter\",
\"id_str\": \"783214\",
\"id\": 783214,
\"indices\": [0, 8],
\"screen_name\": \"twitter\"
},
{
\"name\": \"Picture.ly\",
\"id_str\": \"334715534\",
\"id\": 334715534,
\"indices\": [15, 28],
\"screen_name\": \"SeePicturely\"
},
{
\"name\": \"Bosco So\",
\"id_str\": \"14792670\",
\"id\": 14792670,
\"indices\": [46, 58],
\"screen_name\": \"boscomonkey\"
},
{
\"name\": \"Taylor Singletary\",
\"id_str\": \"819797\",
\"id\": 819797,
\"indices\": [59, 66],
\"screen_name\": \"episod\"
}
]
},
\"in_reply_to_user_id_str\": \"783214\",
\"text\": \"@twitter meets @seepicturely at #tcdisrupt cc.@boscomonkey @episod http://t.co/6J2EgYM\",
\"contributors\": null,
\"id\": 112652479837110273,
\"retweet_count\": 0,
\"in_reply_to_status_id_str\": null,
\"geo\": null,
\"retweeted\": false,
\"possibly_sensitive\": false,
\"in_reply_to_user_id\": 783214,
\"place\": null,
\"source\": \"<a href=\\\"http://instagr.am\\\" rel=\\\"nofollow\\\">Instagram</a>\",
\"user\": {
\"profile_sidebar_border_color\": \"eeeeee\",
\"profile_background_tile\": true,
\"profile_sidebar_fill_color\": \"efefef\",
\"name\": \"Eoin McMillan \",
\"profile_image_url\": \"http://a1.twimg.com/profile_images/1380912173/Screen_shot_2011-06-03_at_7.35.36_PM_normal.png\",
\"created_at\": \"Mon May 16 20:07:59 +0000 2011\",
\"location\": \"Twitter\",
\"profile_link_color\": \"009999\",
\"follow_request_sent\": null,
\"is_translator\": false,
\"id_str\": \"299862462\",
\"favourites_count\": 0,
\"default_profile\": false,
\"url\": \"http://www.eoin.me\",
\"contributors_enabled\": false,
\"id\": 299862462,
\"utc_offset\": null,
\"profile_image_url_https\": \"https://si0.twimg.com/profile_images/1380912173/Screen_shot_2011-06-03_at_7.35.36_PM_normal.png\",
\"profile_use_background_image\": true,
\"listed_count\": 0,
\"followers_count\": 9,
\"lang\": \"en\",
\"profile_text_color\": \"333333\",
\"protected\": false,
\"profile_background_image_url_https\": \"https://si0.twimg.com/images/themes/theme14/bg.gif\",
\"description\": \"Eoin's photography account. See @mceoin for tweets.\",
\"geo_enabled\": false,
\"verified\": false,
\"profile_background_color\": \"131516\",
\"time_zone\": null,
\"notifications\": null,
\"statuses_count\": 255,
\"friends_count\": 0,
\"default_profile_image\": false,
\"profile_background_image_url\": \"http://a1.twimg.com/images/themes/theme14/bg.gif\",
\"screen_name\": \"imeoin\",
\"following\": null,
\"show_all_inline_media\": false
},
\"in_reply_to_screen_name\": \"twitter\",
\"in_reply_to_status_id\": null
}">>.
sample_github_user() ->
<<"{
\"user\": {
\"gravatar_id\": \"b8dbb1987e8e5318584865f880036796\",
\"company\": \"GitHub\",
\"name\": \"Chris Wanstrath\",
\"created_at\": \"2007/10/19 22:24:19 -0700\",
\"location\": \"San Francisco, CA\",
\"public_repo_count\": 98,
\"public_gist_count\": 270,
\"blog\": \"http://chriswanstrath.com/\",
\"following_count\": 196,
\"id\": 2,
\"type\": \"User\",
\"permission\": null,
\"followers_count\": 1692,
\"login\": \"defunkt\",
\"email\": \"chris@wanstrath.com\"
}
}">>.

2
priv/test_cases/bad_low_surrogate_replaced.test
View file

@ -1,4 +1,4 @@
{name, "bad_low_surrogate_replaced"}.
{jsx, [start_array,{string, <<16#fffd/utf8, 16#fffd/utf8>>},end_array,end_json]}.
{json, "bad_low_surrogate_replaced.json"}.
{jsx_flags, [loose_unicode]}.
{jsx_flags, [replaced_bad_utf8]}.

1
priv/test_cases/bom.json Normal file
View file

@ -0,0 +1 @@
[]

3
priv/test_cases/bom.test Normal file
View file

@ -0,0 +1,3 @@
{name, "byte order mark"}.
{jsx, [start_array, end_array, end_json]}.
{json, "bom.json"}.

10
priv/test_cases/comment_style_a.json Normal file
View file

@ -0,0 +1,10 @@
// comment
{ // comment
"key" // comment
: // comment
[ // comment
true // comment
, // comment
false // comment
] // comment
} // comment

4
priv/test_cases/comment_style_a.test Normal file
View file

@ -0,0 +1,4 @@
{name, "comment_style_a"}.
{jsx, [start_object,{key, <<"key">>}, start_array, {literal, true}, {literal, false}, end_array, end_object,end_json]}.
{json, "comment_style_a.json"}.
{jsx_flags, [comments]}.

1
priv/test_cases/comment_style_b.json Normal file
View file

@ -0,0 +1 @@
/* comment */ { /* comment */ "key" /* comment */ : /* comment */ [ /* comment */ true /* comment */ , /* comment */ false /* comment */ ] /* comment */ } /* comment */

4
priv/test_cases/comment_style_b.test Normal file
View file

@ -0,0 +1,4 @@
{name, "comment_style_a"}.
{jsx, [start_object,{key, <<"key">>}, start_array, {literal, true}, {literal, false}, end_array, end_object,end_json]}.
{json, "comment_style_a.json"}.
{jsx_flags, [comments]}.

2
priv/test_cases/unpaired_surrogate_replaced.test
View file

@ -1,4 +1,4 @@
{name, "unpaired surrogate replaced"}.
{jsx, [start_array,{string,<<65533/utf8,$b,$l,$a,$h>>},end_array,end_json]}.
{json, "unpaired_surrogate_replaced.json"}.
{jsx_flags, [loose_unicode]}.
{jsx_flags, [replaced_bad_utf8]}.

13
rebar.config
View file

@ -28,16 +28,3 @@
{xref_checks, [undefined_function_calls]}.
{cover_enabled, true}.
{deps, [
{
'nicedecimal',
".*",
{git, "git://github.com/talentdeficit/nicedecimal.git", {branch, "master"}}
},
{
'meck',
".*",
{git, "git://github.com/eproxus/meck.git", {branch, "master"}}
}
]}.

3
src/jsx.app.src
View file

@ -14,8 +14,7 @@
{registered, []},
{applications, [
kernel,
stdlib,
nicedecimal
stdlib
]},
{env, []}
]}.

16
src/jsx.erl
View file

@ -134,41 +134,41 @@ encoder_decoder_equiv_test_() ->
].
single_quotes_test_() ->
single_quoted_strings_test_() ->
[
{"single quoted keys",
?_assertEqual(
to_term(<<"{'key':true}">>, [single_quotes]),
to_term(<<"{'key':true}">>, [single_quoted_strings]),
[{<<"key">>, true}]
)
},
{"multiple single quoted keys",
?_assertEqual(
to_term(<<"{'key':true, 'another key':true}">>, [single_quotes]),
to_term(<<"{'key':true, 'another key':true}">>, [single_quoted_strings]),
[{<<"key">>, true}, {<<"another key">>, true}]
)
},
{"nested single quoted keys",
?_assertEqual(
to_term(<<"{'key': {'key':true, 'another key':true}}">>, [single_quotes]),
to_term(<<"{'key': {'key':true, 'another key':true}}">>, [single_quoted_strings]),
[{<<"key">>, [{<<"key">>, true}, {<<"another key">>, true}]}]
)
},
{"single quoted string",
?_assertEqual(
to_term(<<"['string']">>, [single_quotes]),
to_term(<<"['string']">>, [single_quoted_strings]),
[<<"string">>]
)
},
{"single quote in double quoted string",
?_assertEqual(
to_term(<<"[\"a single quote: '\"]">>, [single_quotes]),
to_term(<<"[\"a single quote: '\"]">>, [single_quoted_strings]),
[<<"a single quote: '">>]
)
},
{"escaped single quote in single quoted string",
?_assertEqual(
to_term(<<"['a single quote: \\'']">>, [single_quotes]),
to_term(<<"['a single quote: \\'']">>, [single_quoted_strings]),
[<<"a single quote: '">>]
)
},
@ -181,7 +181,7 @@ single_quotes_test_() ->
{"mismatched quotes",
?_assertError(
badarg,
to_term(<<"['mismatched\"]">>, [single_quotes])
to_term(<<"['mismatched\"]">>, [single_quoted_strings])
)
}
].

661
src/jsx_decoder.erl
View file

File diff suppressed because it is too large Load diff

688
src/jsx_encoder.erl
View file

@ -49,7 +49,7 @@ encoder(Handler, State, Opts) ->
start(Term, {Handler, State}, Opts) ->
Handler:handle_event(end_json, value(Term, {Handler, State}, Opts)).
Handler:handle_event(end_json, value(pre_encode(Term, Opts), {Handler, State}, Opts)).
value(String, {Handler, State}, Opts) when is_binary(String) ->
@ -82,7 +82,7 @@ object([{Key, Value}|Rest], {Handler, State}, Opts) ->
{
Handler,
value(
Value,
pre_encode(Value, Opts),
{Handler, Handler:handle_event({key, clean_string(fix_key(Key), Opts)}, State)},
Opts
)
@ -94,112 +94,408 @@ object(Term, Handler, Opts) -> ?error([Term, Handler, Opts]).
list([Value|Rest], {Handler, State}, Opts) ->
list(Rest, {Handler, value(Value, {Handler, State}, Opts)}, Opts);
list(Rest, {Handler, value(pre_encode(Value, Opts), {Handler, State}, Opts)}, Opts);
list([], {Handler, State}, _Opts) -> Handler:handle_event(end_array, State);
list(Term, Handler, Opts) -> ?error([Term, Handler, Opts]).
pre_encode(Value, #opts{pre_encode=false}) -> Value;
pre_encode(Value, Opts) -> (Opts#opts.pre_encode)(Value).
fix_key(Key) when is_atom(Key) -> fix_key(atom_to_binary(Key, utf8));
fix_key(Key) when is_binary(Key) -> Key.
clean_string(Bin, Opts) ->
case Opts#opts.loose_unicode of
true -> maybe_escape(clean_string(Bin, 0, size(Bin), Opts), Opts)
; false ->
case is_clean(Bin) of
true -> maybe_escape(Bin, Opts)
; false -> erlang:error(badarg, [Bin, Opts])
end
case Opts#opts.replaced_bad_utf8 orelse Opts#opts.escaped_strings of
true -> clean(Bin, [], Opts)
; false -> ensure_clean(Bin), Bin
end.
maybe_escape(Bin, Opts=#opts{json_escape=true}) -> jsx_utils:json_escape(Bin, Opts);
maybe_escape(Bin, _) -> Bin.
%% fast path for no escaping and no correcting, throws error if string is 'bad'
ensure_clean(<<>>) -> ok;
ensure_clean(<<0, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<1, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<2, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<3, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<4, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<5, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<6, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<7, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<8, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<9, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<10, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<11, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<12, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<13, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<14, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<15, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<16, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<17, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<18, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<19, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<20, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<21, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<22, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<23, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<24, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<25, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<26, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<27, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<28, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<29, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<30, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<31, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<32, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<33, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<34, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<35, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<36, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<37, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<38, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<39, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<40, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<41, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<42, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<43, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<44, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<45, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<46, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<47, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<48, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<49, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<50, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<51, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<52, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<53, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<54, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<55, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<56, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<57, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<58, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<59, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<60, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<61, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<62, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<63, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<64, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<65, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<66, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<67, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<68, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<69, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<70, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<71, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<72, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<73, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<74, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<75, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<76, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<77, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<78, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<79, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<80, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<81, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<82, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<83, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<84, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<85, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<86, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<87, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<88, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<89, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<90, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<91, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<92, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<93, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<94, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<95, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<96, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<97, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<98, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<99, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<100, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<101, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<102, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<103, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<104, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<105, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<106, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<107, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<108, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<109, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<110, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<111, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<112, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<113, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<114, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<115, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<116, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<117, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<118, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<119, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<120, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<121, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<122, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<123, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<124, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<125, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<126, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<127, Rest/binary>>) -> ensure_clean(Rest);
ensure_clean(<<X/utf8, Rest/binary>>) when X < 16#800 -> ensure_clean(Rest);
ensure_clean(<<X/utf8, Rest/binary>>) when X < 16#dcff -> ensure_clean(Rest);
ensure_clean(<<X/utf8, Rest/binary>>) when X > 16#dfff, X < 16#fdd0 -> ensure_clean(Rest);
ensure_clean(<<X/utf8, Rest/binary>>) when X > 16#fdef, X < 16#fffe -> ensure_clean(Rest);
ensure_clean(<<X/utf8, Rest/binary>>) when X >= 16#10000, X < 16#1fffe -> ensure_clean(Rest);
ensure_clean(<<X/utf8, Rest/binary>>) when X >= 16#20000, X < 16#2fffe -> ensure_clean(Rest);
ensure_clean(<<X/utf8, Rest/binary>>) when X >= 16#30000, X < 16#3fffe -> ensure_clean(Rest);
ensure_clean(<<X/utf8, Rest/binary>>) when X >= 16#40000, X < 16#4fffe -> ensure_clean(Rest);
ensure_clean(<<X/utf8, Rest/binary>>) when X >= 16#50000, X < 16#5fffe -> ensure_clean(Rest);
ensure_clean(<<X/utf8, Rest/binary>>) when X >= 16#60000, X < 16#6fffe -> ensure_clean(Rest);
ensure_clean(<<X/utf8, Rest/binary>>) when X >= 16#70000, X < 16#7fffe -> ensure_clean(Rest);
ensure_clean(<<X/utf8, Rest/binary>>) when X >= 16#80000, X < 16#8fffe -> ensure_clean(Rest);
ensure_clean(<<X/utf8, Rest/binary>>) when X >= 16#90000, X < 16#9fffe -> ensure_clean(Rest);
ensure_clean(<<X/utf8, Rest/binary>>) when X >= 16#a0000, X < 16#afffe -> ensure_clean(Rest);
ensure_clean(<<X/utf8, Rest/binary>>) when X >= 16#b0000, X < 16#bfffe -> ensure_clean(Rest);
ensure_clean(<<X/utf8, Rest/binary>>) when X >= 16#c0000, X < 16#cfffe -> ensure_clean(Rest);
ensure_clean(<<X/utf8, Rest/binary>>) when X >= 16#d0000, X < 16#dfffe -> ensure_clean(Rest);
ensure_clean(<<X/utf8, Rest/binary>>) when X >= 16#e0000, X < 16#efffe -> ensure_clean(Rest);
ensure_clean(<<X/utf8, Rest/binary>>) when X >= 16#f0000, X < 16#ffffe -> ensure_clean(Rest);
ensure_clean(<<X/utf8, Rest/binary>>) when X >= 16#100000, X < 16#10fffe -> ensure_clean(Rest);
ensure_clean(Bin) -> erlang:error(badarg, [Bin]).
is_clean(<<>>) -> true;
is_clean(<<X/utf8, Rest/binary>>) when X < 16#80 -> is_clean(Rest);
is_clean(<<X/utf8, Rest/binary>>) when X < 16#800 -> is_clean(Rest);
is_clean(<<X/utf8, Rest/binary>>) when X < 16#dcff -> is_clean(Rest);
is_clean(<<X/utf8, Rest/binary>>) when X > 16#dfff, X < 16#fdd0 -> is_clean(Rest);
is_clean(<<X/utf8, Rest/binary>>) when X > 16#fdef, X < 16#fffe -> is_clean(Rest);
is_clean(<<X/utf8, Rest/binary>>) when X >= 16#10000, X < 16#1fffe -> is_clean(Rest);
is_clean(<<X/utf8, Rest/binary>>) when X >= 16#20000, X < 16#2fffe -> is_clean(Rest);
is_clean(<<X/utf8, Rest/binary>>) when X >= 16#30000, X < 16#3fffe -> is_clean(Rest);
is_clean(<<X/utf8, Rest/binary>>) when X >= 16#40000, X < 16#4fffe -> is_clean(Rest);
is_clean(<<X/utf8, Rest/binary>>) when X >= 16#50000, X < 16#5fffe -> is_clean(Rest);
is_clean(<<X/utf8, Rest/binary>>) when X >= 16#60000, X < 16#6fffe -> is_clean(Rest);
is_clean(<<X/utf8, Rest/binary>>) when X >= 16#70000, X < 16#7fffe -> is_clean(Rest);
is_clean(<<X/utf8, Rest/binary>>) when X >= 16#80000, X < 16#8fffe -> is_clean(Rest);
is_clean(<<X/utf8, Rest/binary>>) when X >= 16#90000, X < 16#9fffe -> is_clean(Rest);
is_clean(<<X/utf8, Rest/binary>>) when X >= 16#a0000, X < 16#afffe -> is_clean(Rest);
is_clean(<<X/utf8, Rest/binary>>) when X >= 16#b0000, X < 16#bfffe -> is_clean(Rest);
is_clean(<<X/utf8, Rest/binary>>) when X >= 16#c0000, X < 16#cfffe -> is_clean(Rest);
is_clean(<<X/utf8, Rest/binary>>) when X >= 16#d0000, X < 16#dfffe -> is_clean(Rest);
is_clean(<<X/utf8, Rest/binary>>) when X >= 16#e0000, X < 16#efffe -> is_clean(Rest);
is_clean(<<X/utf8, Rest/binary>>) when X >= 16#f0000, X < 16#ffffe -> is_clean(Rest);
is_clean(<<X/utf8, Rest/binary>>) when X >= 16#100000, X < 16#10fffe -> is_clean(Rest);
is_clean(Bin) -> erlang:error(badarg, [Bin]).
clean_string(Str, Len, Len, _Opts) -> Str;
clean_string(Str, L, Len, Opts) ->
case Str of
<<_:L/binary, X/utf8, _/binary>> when X < 16#80 -> clean_string(Str, L + 1, Len, Opts)
; <<_:L/binary, X/utf8, _/binary>> when X < 16#800 -> clean_string(Str, L + 2, Len, Opts)
; <<_:L/binary, X/utf8, _/binary>> when X < 16#dcff -> clean_string(Str, L + 3, Len, Opts)
; <<_:L/binary, X/utf8, _/binary>> when X > 16#dfff, X < 16#fdd0 -> clean_string(Str, L + 3, Len, Opts)
; <<_:L/binary, X/utf8, _/binary>> when X > 16#fdef, X < 16#fffe -> clean_string(Str, L + 3, Len, Opts)
; <<_:L/binary, X/utf8, _/binary>> when X >= 16#10000, X < 16#1fffe -> clean_string(Str, L + 4, Len, Opts)
; <<_:L/binary, X/utf8, _/binary>> when X >= 16#20000, X < 16#2fffe -> clean_string(Str, L + 4, Len, Opts)
; <<_:L/binary, X/utf8, _/binary>> when X >= 16#30000, X < 16#3fffe -> clean_string(Str, L + 4, Len, Opts)
; <<_:L/binary, X/utf8, _/binary>> when X >= 16#40000, X < 16#4fffe -> clean_string(Str, L + 4, Len, Opts)
; <<_:L/binary, X/utf8, _/binary>> when X >= 16#50000, X < 16#5fffe -> clean_string(Str, L + 4, Len, Opts)
; <<_:L/binary, X/utf8, _/binary>> when X >= 16#60000, X < 16#6fffe -> clean_string(Str, L + 4, Len, Opts)
; <<_:L/binary, X/utf8, _/binary>> when X >= 16#70000, X < 16#7fffe -> clean_string(Str, L + 4, Len, Opts)
; <<_:L/binary, X/utf8, _/binary>> when X >= 16#80000, X < 16#8fffe -> clean_string(Str, L + 4, Len, Opts)
; <<_:L/binary, X/utf8, _/binary>> when X >= 16#90000, X < 16#9fffe -> clean_string(Str, L + 4, Len, Opts)
; <<_:L/binary, X/utf8, _/binary>> when X >= 16#a0000, X < 16#afffe -> clean_string(Str, L + 4, Len, Opts)
; <<_:L/binary, X/utf8, _/binary>> when X >= 16#b0000, X < 16#bfffe -> clean_string(Str, L + 4, Len, Opts)
; <<_:L/binary, X/utf8, _/binary>> when X >= 16#c0000, X < 16#cfffe -> clean_string(Str, L + 4, Len, Opts)
; <<_:L/binary, X/utf8, _/binary>> when X >= 16#d0000, X < 16#dfffe -> clean_string(Str, L + 4, Len, Opts)
; <<_:L/binary, X/utf8, _/binary>> when X >= 16#e0000, X < 16#efffe -> clean_string(Str, L + 4, Len, Opts)
; <<_:L/binary, X/utf8, _/binary>> when X >= 16#f0000, X < 16#ffffe -> clean_string(Str, L + 4, Len, Opts)
; <<_:L/binary, X/utf8, _/binary>> when X >= 16#100000, X < 16#10fffe -> clean_string(Str, L + 4, Len, Opts)
%% escape and/or replace bad codepoints if requested
clean(<<>>, Acc, _Opts) -> unicode:characters_to_binary(lists:reverse(Acc));
clean(<<0, Rest/binary>>, Acc, Opts) -> clean(Rest, maybe_replace(0, Opts) ++ Acc, Opts);
clean(<<1, Rest/binary>>, Acc, Opts) -> clean(Rest, maybe_replace(1, Opts) ++ Acc, Opts);
clean(<<2, Rest/binary>>, Acc, Opts) -> clean(Rest, maybe_replace(2, Opts) ++ Acc, Opts);
clean(<<3, Rest/binary>>, Acc, Opts) -> clean(Rest, maybe_replace(3, Opts) ++ Acc, Opts);
clean(<<4, Rest/binary>>, Acc, Opts) -> clean(Rest, maybe_replace(4, Opts) ++ Acc, Opts);
clean(<<5, Rest/binary>>, Acc, Opts) -> clean(Rest, maybe_replace(5, Opts) ++ Acc, Opts);
clean(<<6, Rest/binary>>, Acc, Opts) -> clean(Rest, maybe_replace(6, Opts) ++ Acc, Opts);
clean(<<7, Rest/binary>>, Acc, Opts) -> clean(Rest, maybe_replace(7, Opts) ++ Acc, Opts);
clean(<<8, Rest/binary>>, Acc, Opts) -> clean(Rest, maybe_replace(8, Opts) ++ Acc, Opts);
clean(<<9, Rest/binary>>, Acc, Opts) -> clean(Rest, maybe_replace(9, Opts) ++ Acc, Opts);
clean(<<10, Rest/binary>>, Acc, Opts) -> clean(Rest, maybe_replace(10, Opts) ++ Acc, Opts);
clean(<<11, Rest/binary>>, Acc, Opts) -> clean(Rest, maybe_replace(11, Opts) ++ Acc, Opts);
clean(<<12, Rest/binary>>, Acc, Opts) -> clean(Rest, maybe_replace(12, Opts) ++ Acc, Opts);
clean(<<13, Rest/binary>>, Acc, Opts) -> clean(Rest, maybe_replace(13, Opts) ++ Acc, Opts);
clean(<<14, Rest/binary>>, Acc, Opts) -> clean(Rest, maybe_replace(14, Opts) ++ Acc, Opts);
clean(<<15, Rest/binary>>, Acc, Opts) -> clean(Rest, maybe_replace(15, Opts) ++ Acc, Opts);
clean(<<16, Rest/binary>>, Acc, Opts) -> clean(Rest, maybe_replace(16, Opts) ++ Acc, Opts);
clean(<<17, Rest/binary>>, Acc, Opts) -> clean(Rest, maybe_replace(17, Opts) ++ Acc, Opts);
clean(<<18, Rest/binary>>, Acc, Opts) -> clean(Rest, maybe_replace(18, Opts) ++ Acc, Opts);
clean(<<19, Rest/binary>>, Acc, Opts) -> clean(Rest, maybe_replace(19, Opts) ++ Acc, Opts);
clean(<<20, Rest/binary>>, Acc, Opts) -> clean(Rest, maybe_replace(20, Opts) ++ Acc, Opts);
clean(<<21, Rest/binary>>, Acc, Opts) -> clean(Rest, maybe_replace(21, Opts) ++ Acc, Opts);
clean(<<22, Rest/binary>>, Acc, Opts) -> clean(Rest, maybe_replace(22, Opts) ++ Acc, Opts);
clean(<<23, Rest/binary>>, Acc, Opts) -> clean(Rest, maybe_replace(23, Opts) ++ Acc, Opts);
clean(<<24, Rest/binary>>, Acc, Opts) -> clean(Rest, maybe_replace(24, Opts) ++ Acc, Opts);
clean(<<25, Rest/binary>>, Acc, Opts) -> clean(Rest, maybe_replace(25, Opts) ++ Acc, Opts);
clean(<<26, Rest/binary>>, Acc, Opts) -> clean(Rest, maybe_replace(26, Opts) ++ Acc, Opts);
clean(<<27, Rest/binary>>, Acc, Opts) -> clean(Rest, maybe_replace(27, Opts) ++ Acc, Opts);
clean(<<28, Rest/binary>>, Acc, Opts) -> clean(Rest, maybe_replace(28, Opts) ++ Acc, Opts);
clean(<<29, Rest/binary>>, Acc, Opts) -> clean(Rest, maybe_replace(29, Opts) ++ Acc, Opts);
clean(<<30, Rest/binary>>, Acc, Opts) -> clean(Rest, maybe_replace(30, Opts) ++ Acc, Opts);
clean(<<31, Rest/binary>>, Acc, Opts) -> clean(Rest, maybe_replace(31, Opts) ++ Acc, Opts);
clean(<<32, Rest/binary>>, Acc, Opts) -> clean(Rest, [32] ++ Acc, Opts);
clean(<<33, Rest/binary>>, Acc, Opts) -> clean(Rest, [33] ++ Acc, Opts);
clean(<<34, Rest/binary>>, Acc, Opts) -> clean(Rest, maybe_replace(34, Opts) ++ Acc, Opts);
clean(<<35, Rest/binary>>, Acc, Opts) -> clean(Rest, [35] ++ Acc, Opts);
clean(<<36, Rest/binary>>, Acc, Opts) -> clean(Rest, [36] ++ Acc, Opts);
clean(<<37, Rest/binary>>, Acc, Opts) -> clean(Rest, [37] ++ Acc, Opts);
clean(<<38, Rest/binary>>, Acc, Opts) -> clean(Rest, [38] ++ Acc, Opts);
clean(<<39, Rest/binary>>, Acc, Opts) -> clean(Rest, maybe_replace(39, Opts) ++ Acc, Opts);
clean(<<40, Rest/binary>>, Acc, Opts) -> clean(Rest, [40] ++ Acc, Opts);
clean(<<41, Rest/binary>>, Acc, Opts) -> clean(Rest, [41] ++ Acc, Opts);
clean(<<42, Rest/binary>>, Acc, Opts) -> clean(Rest, [42] ++ Acc, Opts);
clean(<<43, Rest/binary>>, Acc, Opts) -> clean(Rest, [43] ++ Acc, Opts);
clean(<<44, Rest/binary>>, Acc, Opts) -> clean(Rest, [44] ++ Acc, Opts);
clean(<<45, Rest/binary>>, Acc, Opts) -> clean(Rest, [45] ++ Acc, Opts);
clean(<<46, Rest/binary>>, Acc, Opts) -> clean(Rest, [46] ++ Acc, Opts);
clean(<<47, Rest/binary>>, Acc, Opts) -> clean(Rest, maybe_replace(47, Opts) ++ Acc, Opts);
clean(<<48, Rest/binary>>, Acc, Opts) -> clean(Rest, [48] ++ Acc, Opts);
clean(<<49, Rest/binary>>, Acc, Opts) -> clean(Rest, [49] ++ Acc, Opts);
clean(<<50, Rest/binary>>, Acc, Opts) -> clean(Rest, [50] ++ Acc, Opts);
clean(<<51, Rest/binary>>, Acc, Opts) -> clean(Rest, [51] ++ Acc, Opts);
clean(<<52, Rest/binary>>, Acc, Opts) -> clean(Rest, [52] ++ Acc, Opts);
clean(<<53, Rest/binary>>, Acc, Opts) -> clean(Rest, [53] ++ Acc, Opts);
clean(<<54, Rest/binary>>, Acc, Opts) -> clean(Rest, [54] ++ Acc, Opts);
clean(<<55, Rest/binary>>, Acc, Opts) -> clean(Rest, [55] ++ Acc, Opts);
clean(<<56, Rest/binary>>, Acc, Opts) -> clean(Rest, [56] ++ Acc, Opts);
clean(<<57, Rest/binary>>, Acc, Opts) -> clean(Rest, [57] ++ Acc, Opts);
clean(<<58, Rest/binary>>, Acc, Opts) -> clean(Rest, [58] ++ Acc, Opts);
clean(<<59, Rest/binary>>, Acc, Opts) -> clean(Rest, [59] ++ Acc, Opts);
clean(<<60, Rest/binary>>, Acc, Opts) -> clean(Rest, [60] ++ Acc, Opts);
clean(<<61, Rest/binary>>, Acc, Opts) -> clean(Rest, [61] ++ Acc, Opts);
clean(<<62, Rest/binary>>, Acc, Opts) -> clean(Rest, [62] ++ Acc, Opts);
clean(<<63, Rest/binary>>, Acc, Opts) -> clean(Rest, [63] ++ Acc, Opts);
clean(<<64, Rest/binary>>, Acc, Opts) -> clean(Rest, [64] ++ Acc, Opts);
clean(<<65, Rest/binary>>, Acc, Opts) -> clean(Rest, [65] ++ Acc, Opts);
clean(<<66, Rest/binary>>, Acc, Opts) -> clean(Rest, [66] ++ Acc, Opts);
clean(<<67, Rest/binary>>, Acc, Opts) -> clean(Rest, [67] ++ Acc, Opts);
clean(<<68, Rest/binary>>, Acc, Opts) -> clean(Rest, [68] ++ Acc, Opts);
clean(<<69, Rest/binary>>, Acc, Opts) -> clean(Rest, [69] ++ Acc, Opts);
clean(<<70, Rest/binary>>, Acc, Opts) -> clean(Rest, [70] ++ Acc, Opts);
clean(<<71, Rest/binary>>, Acc, Opts) -> clean(Rest, [71] ++ Acc, Opts);
clean(<<72, Rest/binary>>, Acc, Opts) -> clean(Rest, [72] ++ Acc, Opts);
clean(<<73, Rest/binary>>, Acc, Opts) -> clean(Rest, [73] ++ Acc, Opts);
clean(<<74, Rest/binary>>, Acc, Opts) -> clean(Rest, [74] ++ Acc, Opts);
clean(<<75, Rest/binary>>, Acc, Opts) -> clean(Rest, [75] ++ Acc, Opts);
clean(<<76, Rest/binary>>, Acc, Opts) -> clean(Rest, [76] ++ Acc, Opts);
clean(<<77, Rest/binary>>, Acc, Opts) -> clean(Rest, [77] ++ Acc, Opts);
clean(<<78, Rest/binary>>, Acc, Opts) -> clean(Rest, [78] ++ Acc, Opts);
clean(<<79, Rest/binary>>, Acc, Opts) -> clean(Rest, [79] ++ Acc, Opts);
clean(<<80, Rest/binary>>, Acc, Opts) -> clean(Rest, [80] ++ Acc, Opts);
clean(<<81, Rest/binary>>, Acc, Opts) -> clean(Rest, [81] ++ Acc, Opts);
clean(<<82, Rest/binary>>, Acc, Opts) -> clean(Rest, [82] ++ Acc, Opts);
clean(<<83, Rest/binary>>, Acc, Opts) -> clean(Rest, [83] ++ Acc, Opts);
clean(<<84, Rest/binary>>, Acc, Opts) -> clean(Rest, [84] ++ Acc, Opts);
clean(<<85, Rest/binary>>, Acc, Opts) -> clean(Rest, [85] ++ Acc, Opts);
clean(<<86, Rest/binary>>, Acc, Opts) -> clean(Rest, [86] ++ Acc, Opts);
clean(<<87, Rest/binary>>, Acc, Opts) -> clean(Rest, [87] ++ Acc, Opts);
clean(<<88, Rest/binary>>, Acc, Opts) -> clean(Rest, [88] ++ Acc, Opts);
clean(<<89, Rest/binary>>, Acc, Opts) -> clean(Rest, [89] ++ Acc, Opts);
clean(<<90, Rest/binary>>, Acc, Opts) -> clean(Rest, [90] ++ Acc, Opts);
clean(<<91, Rest/binary>>, Acc, Opts) -> clean(Rest, [91] ++ Acc, Opts);
clean(<<92, Rest/binary>>, Acc, Opts) -> clean(Rest, maybe_replace(92, Opts) ++ Acc, Opts);
clean(<<93, Rest/binary>>, Acc, Opts) -> clean(Rest, [93] ++ Acc, Opts);
clean(<<94, Rest/binary>>, Acc, Opts) -> clean(Rest, [94] ++ Acc, Opts);
clean(<<95, Rest/binary>>, Acc, Opts) -> clean(Rest, [95] ++ Acc, Opts);
clean(<<96, Rest/binary>>, Acc, Opts) -> clean(Rest, [96] ++ Acc, Opts);
clean(<<97, Rest/binary>>, Acc, Opts) -> clean(Rest, [97] ++ Acc, Opts);
clean(<<98, Rest/binary>>, Acc, Opts) -> clean(Rest, [98] ++ Acc, Opts);
clean(<<99, Rest/binary>>, Acc, Opts) -> clean(Rest, [99] ++ Acc, Opts);
clean(<<100, Rest/binary>>, Acc, Opts) -> clean(Rest, [100] ++ Acc, Opts);
clean(<<101, Rest/binary>>, Acc, Opts) -> clean(Rest, [101] ++ Acc, Opts);
clean(<<102, Rest/binary>>, Acc, Opts) -> clean(Rest, [102] ++ Acc, Opts);
clean(<<103, Rest/binary>>, Acc, Opts) -> clean(Rest, [103] ++ Acc, Opts);
clean(<<104, Rest/binary>>, Acc, Opts) -> clean(Rest, [104] ++ Acc, Opts);
clean(<<105, Rest/binary>>, Acc, Opts) -> clean(Rest, [105] ++ Acc, Opts);
clean(<<106, Rest/binary>>, Acc, Opts) -> clean(Rest, [106] ++ Acc, Opts);
clean(<<107, Rest/binary>>, Acc, Opts) -> clean(Rest, [107] ++ Acc, Opts);
clean(<<108, Rest/binary>>, Acc, Opts) -> clean(Rest, [108] ++ Acc, Opts);
clean(<<109, Rest/binary>>, Acc, Opts) -> clean(Rest, [109] ++ Acc, Opts);
clean(<<110, Rest/binary>>, Acc, Opts) -> clean(Rest, [110] ++ Acc, Opts);
clean(<<111, Rest/binary>>, Acc, Opts) -> clean(Rest, [111] ++ Acc, Opts);
clean(<<112, Rest/binary>>, Acc, Opts) -> clean(Rest, [112] ++ Acc, Opts);
clean(<<113, Rest/binary>>, Acc, Opts) -> clean(Rest, [113] ++ Acc, Opts);
clean(<<114, Rest/binary>>, Acc, Opts) -> clean(Rest, [114] ++ Acc, Opts);
clean(<<115, Rest/binary>>, Acc, Opts) -> clean(Rest, [115] ++ Acc, Opts);
clean(<<116, Rest/binary>>, Acc, Opts) -> clean(Rest, [116] ++ Acc, Opts);
clean(<<117, Rest/binary>>, Acc, Opts) -> clean(Rest, [117] ++ Acc, Opts);
clean(<<118, Rest/binary>>, Acc, Opts) -> clean(Rest, [118] ++ Acc, Opts);
clean(<<119, Rest/binary>>, Acc, Opts) -> clean(Rest, [119] ++ Acc, Opts);
clean(<<120, Rest/binary>>, Acc, Opts) -> clean(Rest, [120] ++ Acc, Opts);
clean(<<121, Rest/binary>>, Acc, Opts) -> clean(Rest, [121] ++ Acc, Opts);
clean(<<122, Rest/binary>>, Acc, Opts) -> clean(Rest, [122] ++ Acc, Opts);
clean(<<123, Rest/binary>>, Acc, Opts) -> clean(Rest, [123] ++ Acc, Opts);
clean(<<124, Rest/binary>>, Acc, Opts) -> clean(Rest, [124] ++ Acc, Opts);
clean(<<125, Rest/binary>>, Acc, Opts) -> clean(Rest, [125] ++ Acc, Opts);
clean(<<126, Rest/binary>>, Acc, Opts) -> clean(Rest, [126] ++ Acc, Opts);
clean(<<127, Rest/binary>>, Acc, Opts) -> clean(Rest, [127] ++ Acc, Opts);
clean(<<X/utf8, Rest/binary>>, Acc, Opts) when X < 16#800 ->
clean(Rest, [X] ++ Acc, Opts);
clean(<<X/utf8, Rest/binary>>, Acc, Opts) when X == 16#2028; X == 16#2029 ->
clean(Rest, maybe_replace(X, Opts) ++ Acc, Opts);
clean(<<X/utf8, Rest/binary>>, Acc, Opts) when X < 16#dcff ->
clean(Rest, [X] ++ Acc, Opts);
clean(<<X/utf8, Rest/binary>>, Acc, Opts) when X > 16#dfff, X < 16#fdd0 ->
clean(Rest, [X] ++ Acc, Opts);
clean(<<X/utf8, Rest/binary>>, Acc, Opts) when X > 16#fdef, X < 16#fffe ->
clean(Rest, [X] ++ Acc, Opts);
clean(<<X/utf8, Rest/binary>>, Acc, Opts) when X >= 16#10000, X < 16#1fffe ->
clean(Rest, [X] ++ Acc, Opts);
clean(<<X/utf8, Rest/binary>>, Acc, Opts) when X >= 16#20000, X < 16#2fffe ->
clean(Rest, [X] ++ Acc, Opts);
clean(<<X/utf8, Rest/binary>>, Acc, Opts) when X >= 16#30000, X < 16#3fffe ->
clean(Rest, [X] ++ Acc, Opts);
clean(<<X/utf8, Rest/binary>>, Acc, Opts) when X >= 16#40000, X < 16#4fffe ->
clean(Rest, [X] ++ Acc, Opts);
clean(<<X/utf8, Rest/binary>>, Acc, Opts) when X >= 16#50000, X < 16#5fffe ->
clean(Rest, [X] ++ Acc, Opts);
clean(<<X/utf8, Rest/binary>>, Acc, Opts) when X >= 16#60000, X < 16#6fffe ->
clean(Rest, [X] ++ Acc, Opts);
clean(<<X/utf8, Rest/binary>>, Acc, Opts) when X >= 16#70000, X < 16#7fffe ->
clean(Rest, [X] ++ Acc, Opts);
clean(<<X/utf8, Rest/binary>>, Acc, Opts) when X >= 16#80000, X < 16#8fffe ->
clean(Rest, [X] ++ Acc, Opts);
clean(<<X/utf8, Rest/binary>>, Acc, Opts) when X >= 16#90000, X < 16#9fffe ->
clean(Rest, [X] ++ Acc, Opts);
clean(<<X/utf8, Rest/binary>>, Acc, Opts) when X >= 16#a0000, X < 16#afffe ->
clean(Rest, [X] ++ Acc, Opts);
clean(<<X/utf8, Rest/binary>>, Acc, Opts) when X >= 16#b0000, X < 16#bfffe ->
clean(Rest, [X] ++ Acc, Opts);
clean(<<X/utf8, Rest/binary>>, Acc, Opts) when X >= 16#c0000, X < 16#cfffe ->
clean(Rest, [X] ++ Acc, Opts);
clean(<<X/utf8, Rest/binary>>, Acc, Opts) when X >= 16#d0000, X < 16#dfffe ->
clean(Rest, [X] ++ Acc, Opts);
clean(<<X/utf8, Rest/binary>>, Acc, Opts) when X >= 16#e0000, X < 16#efffe ->
clean(Rest, [X] ++ Acc, Opts);
clean(<<X/utf8, Rest/binary>>, Acc, Opts) when X >= 16#f0000, X < 16#ffffe ->
clean(Rest, [X] ++ Acc, Opts);
clean(<<X/utf8, Rest/binary>>, Acc, Opts) when X >= 16#100000, X < 16#10fffe ->
clean(Rest, [X] ++ Acc, Opts);
%% noncharacters
; <<H:L/binary, X/utf8, T/binary>> when X < 16#10000 ->
clean_string(<<H:L/binary, 16#fffd/utf8, T/binary>>, L + 3, Len, Opts)
; <<H:L/binary, _/utf8, T/binary>> ->
clean_string(<<H:L/binary, 16#fffd/utf8, T/binary>>, L + 4, Len, Opts)
clean(<<_/utf8, Rest/binary>>, Acc, Opts) ->
clean(Rest, maybe_replace(noncharacter, Opts) ++ Acc, Opts);
%% surrogates
; <<H:L/binary, 237, X, _, T/binary>> when X >= 160 ->
clean_string(<<H:L/binary, 16#fffd/utf8, T/binary>>, L + 3, Len, Opts)
clean(<<237, X, _, Rest/binary>>, Acc, Opts) when X >= 160 ->
clean(Rest, maybe_replace(surrogate, Opts) ++ Acc, Opts);
%% u+fffe and u+ffff for R14BXX
; <<H:L/binary, 239, 191, X, T/binary>> when X == 190; X == 191 ->
clean_string(<<H:L/binary, 16#fffd/utf8, T/binary>>, L + 3, Len, Opts)
clean(<<239, 191, X, Rest/binary>>, Acc, Opts) when X == 190; X == 191 ->
clean(Rest, maybe_replace(noncharacter, Opts) ++ Acc, Opts);
%% overlong encodings and missing continuations of a 2 byte sequence
; <<H:L/binary, X, T/binary>> when X >= 192, X =< 223 ->
{Tail, Stripped} = strip_continuations(T, 1, 0),
clean_string(<<H:L/binary, 16#fffd/utf8, Tail/binary>>, L + 3, Len + 2 - Stripped, Opts)
clean(<<X, Rest/binary>>, Acc, Opts) when X >= 192, X =< 223 ->
clean(strip_continuations(Rest, 1), maybe_replace(badutf, Opts) ++ Acc, Opts);
%% overlong encodings and missing continuations of a 3 byte sequence
; <<H:L/binary, X, T/binary>> when X >= 224, X =< 239 ->
{Tail, Stripped} = strip_continuations(T, 2, 0),
clean_string(<<H:L/binary, 16#fffd/utf8, Tail/binary>>, L + 3, Len + 2 - Stripped, Opts)
clean(<<X, Rest/binary>>, Acc, Opts) when X >= 224, X =< 239 ->
clean(strip_continuations(Rest, 2), maybe_replace(badutf, Opts) ++ Acc, Opts);
%% overlong encodings and missing continuations of a 4 byte sequence
; <<H:L/binary, X, T/binary>> when X >= 240, X =< 247 ->
{Tail, Stripped} = strip_continuations(T, 3, 0),
clean_string(<<H:L/binary, 16#fffd/utf8, Tail/binary>>, L + 3, Len + 2 - Stripped, Opts)
; <<H:L/binary, _, T/binary>> ->
clean_string(<<H:L/binary, 16#fffd/utf8, T/binary>>, L + 3, Len + 2, Opts)
end.
clean(<<X, Rest/binary>>, Acc, Opts) when X >= 240, X =< 247 ->
clean(strip_continuations(Rest, 3), maybe_replace(badutf, Opts) ++ Acc, Opts);
clean(<<_, Rest/binary>>, Acc, Opts) ->
clean(Rest, maybe_replace(badutf, Opts) ++ Acc, Opts).
strip_continuations(Bin, 0, N) -> {Bin, N};
strip_continuations(<<X, Rest/binary>>, N, M) when X >= 128, X =< 191 ->
strip_continuations(Rest, N - 1, M + 1);
strip_continuations(Bin, 0) -> Bin;
strip_continuations(<<X, Rest/binary>>, N) when X >= 128, X =< 191 ->
strip_continuations(Rest, N - 1);
%% not a continuation byte
strip_continuations(Bin, _, N) -> {Bin, N}.
strip_continuations(Bin, _) -> Bin.
maybe_replace(X, #opts{dirty_strings=true}) when is_integer(X) -> [X];
maybe_replace($\b, #opts{escaped_strings=true}) -> [$b, $\\];
maybe_replace($\t, #opts{escaped_strings=true}) -> [$t, $\\];
maybe_replace($\n, #opts{escaped_strings=true}) -> [$n, $\\];
maybe_replace($\f, #opts{escaped_strings=true}) -> [$f, $\\];
maybe_replace($\r, #opts{escaped_strings=true}) -> [$r, $\\];
maybe_replace($\", #opts{escaped_strings=true}) -> [$\", $\\];
maybe_replace($', Opts=#opts{escaped_strings=true}) ->
case Opts#opts.single_quoted_strings of
true -> [$', $\\]
; false -> [$']
end;
maybe_replace($/, Opts=#opts{escaped_strings=true}) ->
case Opts#opts.escaped_forward_slashes of
true -> [$/, $\\]
; false -> [$/]
end;
maybe_replace($\\, #opts{escaped_strings=true}) -> [$\\, $\\];
maybe_replace(X, Opts=#opts{escaped_strings=true}) when X == 16#2028; X == 16#2029 ->
case Opts#opts.unescaped_jsonp of
true -> [X]
; false -> lists:reverse(jsx_utils:json_escape_sequence(X))
end;
maybe_replace(X, #opts{escaped_strings=true}) when X < 32 ->
lists:reverse(jsx_utils:json_escape_sequence(X));
maybe_replace(noncharacter, #opts{replaced_bad_utf8=true}) -> [16#fffd];
maybe_replace(surrogate, #opts{replaced_bad_utf8=true}) -> [16#fffd];
maybe_replace(badutf, #opts{replaced_bad_utf8=true}) -> [16#fffd].
-ifdef(TEST).
@ -208,7 +504,7 @@ strip_continuations(Bin, _, N) -> {Bin, N}.
xcode(Bin) -> xcode(Bin, #opts{}).
xcode(Bin, [loose_unicode]) -> xcode(Bin, #opts{loose_unicode=true});
xcode(Bin, [replaced_bad_utf8]) -> xcode(Bin, #opts{replaced_bad_utf8=true});
xcode(Bin, Opts) ->
try clean_string(Bin, Opts)
catch error:badarg -> {error, badarg}
@ -225,20 +521,20 @@ bad_utf8_test_() ->
?_assert(is_bad(xcode(<<16#0080>>)))
},
{"orphan continuation byte u+0080 replaced",
?_assertEqual(xcode(<<16#0080>>, [loose_unicode]), <<16#fffd/utf8>>)
?_assertEqual(xcode(<<16#0080>>, [replaced_bad_utf8]), <<16#fffd/utf8>>)
},
{"orphan continuation byte u+00bf",
?_assert(is_bad(xcode(<<16#00bf>>)))
},
{"orphan continuation byte u+00bf replaced",
?_assertEqual(xcode(<<16#00bf>>, [loose_unicode]), <<16#fffd/utf8>>)
?_assertEqual(xcode(<<16#00bf>>, [replaced_bad_utf8]), <<16#fffd/utf8>>)
},
{"2 continuation bytes",
?_assert(is_bad(xcode(<<(binary:copy(<<16#0080>>, 2))/binary>>)))
},
{"2 continuation bytes replaced",
?_assertEqual(
xcode(<<(binary:copy(<<16#0080>>, 2))/binary>>, [loose_unicode]),
xcode(<<(binary:copy(<<16#0080>>, 2))/binary>>, [replaced_bad_utf8]),
binary:copy(<<16#fffd/utf8>>, 2)
)
},
@ -247,7 +543,7 @@ bad_utf8_test_() ->
},
{"3 continuation bytes replaced",
?_assertEqual(
xcode(<<(binary:copy(<<16#0080>>, 3))/binary>>, [loose_unicode]),
xcode(<<(binary:copy(<<16#0080>>, 3))/binary>>, [replaced_bad_utf8]),
binary:copy(<<16#fffd/utf8>>, 3)
)
},
@ -256,7 +552,7 @@ bad_utf8_test_() ->
},
{"4 continuation bytes replaced",
?_assertEqual(
xcode(<<(binary:copy(<<16#0080>>, 4))/binary>>, [loose_unicode]),
xcode(<<(binary:copy(<<16#0080>>, 4))/binary>>, [replaced_bad_utf8]),
binary:copy(<<16#fffd/utf8>>, 4)
)
},
@ -265,7 +561,7 @@ bad_utf8_test_() ->
},
{"5 continuation bytes replaced",
?_assertEqual(
xcode(<<(binary:copy(<<16#0080>>, 5))/binary>>, [loose_unicode]),
xcode(<<(binary:copy(<<16#0080>>, 5))/binary>>, [replaced_bad_utf8]),
binary:copy(<<16#fffd/utf8>>, 5)
)
},
@ -274,7 +570,7 @@ bad_utf8_test_() ->
},
{"6 continuation bytes replaced",
?_assertEqual(
xcode(<<(binary:copy(<<16#0080>>, 6))/binary>>, [loose_unicode]),
xcode(<<(binary:copy(<<16#0080>>, 6))/binary>>, [replaced_bad_utf8]),
binary:copy(<<16#fffd/utf8>>, 6)
)
},
@ -283,7 +579,7 @@ bad_utf8_test_() ->
},
{"all continuation bytes replaced",
?_assertEqual(
xcode(<<(list_to_binary(lists:seq(16#0080, 16#00bf)))/binary>>, [loose_unicode]),
xcode(<<(list_to_binary(lists:seq(16#0080, 16#00bf)))/binary>>, [replaced_bad_utf8]),
binary:copy(<<16#fffd/utf8>>, length(lists:seq(16#0080, 16#00bf)))
)
},
@ -292,7 +588,7 @@ bad_utf8_test_() ->
},
{"lonely start byte replaced",
?_assertEqual(
xcode(<<16#00c0>>, [loose_unicode]),
xcode(<<16#00c0>>, [replaced_bad_utf8]),
<<16#fffd/utf8>>
)
},
@ -301,7 +597,7 @@ bad_utf8_test_() ->
},
{"lonely start bytes (2 byte) replaced",
?_assertEqual(
xcode(<<16#00c0, 32, 16#00df>>, [loose_unicode]),
xcode(<<16#00c0, 32, 16#00df>>, [replaced_bad_utf8]),
<<16#fffd/utf8, 32, 16#fffd/utf8>>
)
},
@ -310,7 +606,7 @@ bad_utf8_test_() ->
},
{"lonely start bytes (3 byte) replaced",
?_assertEqual(
xcode(<<16#00e0, 32, 16#00ef>>, [loose_unicode]),
xcode(<<16#00e0, 32, 16#00ef>>, [replaced_bad_utf8]),
<<16#fffd/utf8, 32, 16#fffd/utf8>>
)
},
@ -319,7 +615,7 @@ bad_utf8_test_() ->
},
{"lonely start bytes (4 byte) replaced",
?_assertEqual(
xcode(<<16#00f0, 32, 16#00f7>>, [loose_unicode]),
xcode(<<16#00f0, 32, 16#00f7>>, [replaced_bad_utf8]),
<<16#fffd/utf8, 32, 16#fffd/utf8>>
)
},
@ -328,25 +624,25 @@ bad_utf8_test_() ->
},
{"missing continuation byte (3 byte) replaced",
?_assertEqual(
xcode(<<224, 160, 32>>, [loose_unicode]),
xcode(<<224, 160, 32>>, [replaced_bad_utf8]),
<<16#fffd/utf8, 32>>
)
},
{"missing continuation byte (4 byte missing one)",
?_assert(is_bad(xcode(<<240, 144, 128, 32>>)))
},
{"missing continuation byte2 (4 byte missing one) replaced",
{"missing continuation byte (4 byte missing one) replaced",
?_assertEqual(
xcode(<<240, 144, 128, 32>>, [loose_unicode]),
xcode(<<240, 144, 128, 32>>, [replaced_bad_utf8]),
<<16#fffd/utf8, 32>>
)
},
{"missing continuation byte (4 byte missing two)",
?_assert(is_bad(xcode(<<240, 144, 32>>)))
},
{"missing continuation byte2 (4 byte missing two) replaced",
{"missing continuation byte (4 byte missing two) replaced",
?_assertEqual(
xcode(<<240, 144, 32>>, [loose_unicode]),
xcode(<<240, 144, 32>>, [replaced_bad_utf8]),
<<16#fffd/utf8, 32>>
)
},
@ -355,7 +651,7 @@ bad_utf8_test_() ->
},
{"overlong encoding of u+002f (2 byte) replaced",
?_assertEqual(
xcode(<<16#c0, 16#af, 32>>, [loose_unicode]),
xcode(<<16#c0, 16#af, 32>>, [replaced_bad_utf8]),
<<16#fffd/utf8, 32>>
)
},
@ -364,7 +660,7 @@ bad_utf8_test_() ->
},
{"overlong encoding of u+002f (3 byte) replaced",
?_assertEqual(
xcode(<<16#e0, 16#80, 16#af, 32>>, [loose_unicode]),
xcode(<<16#e0, 16#80, 16#af, 32>>, [replaced_bad_utf8]),
<<16#fffd/utf8, 32>>
)
},
@ -373,7 +669,7 @@ bad_utf8_test_() ->
},
{"overlong encoding of u+002f (4 byte) replaced",
?_assertEqual(
xcode(<<16#f0, 16#80, 16#80, 16#af, 32>>, [loose_unicode]),
xcode(<<16#f0, 16#80, 16#80, 16#af, 32>>, [replaced_bad_utf8]),
<<16#fffd/utf8, 32>>
)
},
@ -382,7 +678,7 @@ bad_utf8_test_() ->
},
{"highest overlong 2 byte sequence replaced",
?_assertEqual(
xcode(<<16#c1, 16#bf, 32>>, [loose_unicode]),
xcode(<<16#c1, 16#bf, 32>>, [replaced_bad_utf8]),
<<16#fffd/utf8, 32>>
)
},
@ -391,7 +687,7 @@ bad_utf8_test_() ->
},
{"highest overlong 3 byte sequence replaced",
?_assertEqual(
xcode(<<16#e0, 16#9f, 16#bf, 32>>, [loose_unicode]),
xcode(<<16#e0, 16#9f, 16#bf, 32>>, [replaced_bad_utf8]),
<<16#fffd/utf8, 32>>
)
},
@ -400,7 +696,7 @@ bad_utf8_test_() ->
},
{"highest overlong 4 byte sequence replaced",
?_assertEqual(
xcode(<<16#f0, 16#8f, 16#bf, 16#bf, 32>>, [loose_unicode]),
xcode(<<16#f0, 16#8f, 16#bf, 16#bf, 32>>, [replaced_bad_utf8]),
<<16#fffd/utf8, 32>>
)
}
@ -418,7 +714,7 @@ encode(Term, Opts) ->
encode_test_() ->
[
{"naked string", ?_assertEqual(encode(<<"a string\n">>), [{string, <<"a string\n">>}, end_json])},
{"escaped naked string", ?_assertEqual(encode(<<"a string\n">>, [json_escape]), [{string, <<"a string\\n">>}, end_json])},
{"escaped naked string", ?_assertEqual(encode(<<"a string\n">>, [escaped_strings]), [{string, <<"a string\\n">>}, end_json])},
{"naked integer", ?_assertEqual(encode(123), [{integer, 123}, end_json])},
{"naked float", ?_assertEqual(encode(1.23), [{float, 1.23}, end_json])},
{"naked literal", ?_assertEqual(encode(null), [{literal, null}, end_json])},
@ -488,13 +784,130 @@ encode_test_() ->
].
pre_encoders_test_() ->
Term = [
{<<"object">>, [
{<<"literals">>, [true, false, null]},
{<<"strings">>, [<<"foo">>, <<"bar">>, <<"baz">>]},
{<<"numbers">>, [1, 1.0, 1.0e0]}
]}
],
[
{"no pre encode", ?_assertEqual(
encode(Term, []),
[
start_object,
{key, <<"object">>}, start_object,
{key, <<"literals">>}, start_array,
{literal, true}, {literal, false}, {literal, null},
end_array,
{key, <<"strings">>}, start_array,
{string, <<"foo">>}, {string, <<"bar">>}, {string, <<"baz">>},
end_array,
{key, <<"numbers">>}, start_array,
{integer, 1}, {float, 1.0}, {float, 1.0},
end_array,
end_object,
end_object,
end_json
]
)},
{"replace lists with empty lists", ?_assertEqual(
encode(Term, [{pre_encode, fun(V) -> case V of [{_,_}|_] -> V; [{}] -> V; V when is_list(V) -> []; _ -> V end end}]),
[
start_object,
{key, <<"object">>}, start_object,
{key, <<"literals">>}, start_array, end_array,
{key, <<"strings">>}, start_array, end_array,
{key, <<"numbers">>}, start_array, end_array,
end_object,
end_object,
end_json
]
)},
{"replace objects with empty objects", ?_assertEqual(
encode(Term, [{pre_encode, fun(V) -> case V of [{_,_}|_] -> [{}]; _ -> V end end}]),
[
start_object,
end_object,
end_json
]
)},
{"replace all non-list values with false", ?_assertEqual(
encode(Term, [{pre_encode, fun(V) when is_list(V) -> V; (_) -> false end}]),
[
start_object,
{key, <<"object">>}, start_object,
{key, <<"literals">>}, start_array,
{literal, false}, {literal, false}, {literal, false},
end_array,
{key, <<"strings">>}, start_array,
{literal, false}, {literal, false}, {literal, false},
end_array,
{key, <<"numbers">>}, start_array,
{literal, false}, {literal, false}, {literal, false},
end_array,
end_object,
end_object,
end_json
]
)},
{"replace all atoms with atom_to_list", ?_assertEqual(
encode(Term, [{pre_encode, fun(V) when is_atom(V) -> unicode:characters_to_binary(atom_to_list(V)); (V) -> V end}]),
[
start_object,
{key, <<"object">>}, start_object,
{key, <<"literals">>}, start_array,
{string, <<"true">>}, {string, <<"false">>}, {string, <<"null">>},
end_array,
{key, <<"strings">>}, start_array,
{string, <<"foo">>}, {string, <<"bar">>}, {string, <<"baz">>},
end_array,
{key, <<"numbers">>}, start_array,
{integer, 1}, {float, 1.0}, {float, 1.0},
end_array,
end_object,
end_object,
end_json
]
)}
].
escapes_test_() ->
[
{"backspace escape", ?_assertEqual(encode(<<"\b">>, [escaped_strings]), [{string, <<"\\b">>}, end_json])},
{"formfeed escape", ?_assertEqual(encode(<<"\f">>, [escaped_strings]), [{string, <<"\\f">>}, end_json])},
{"newline escape", ?_assertEqual(encode(<<"\n">>, [escaped_strings]), [{string, <<"\\n">>}, end_json])},
{"carriage return escape", ?_assertEqual(encode(<<"\r">>, [escaped_strings]), [{string, <<"\\r">>}, end_json])},
{"tab escape", ?_assertEqual(encode(<<"\t">>, [escaped_strings]), [{string, <<"\\t">>}, end_json])},
{"quote escape", ?_assertEqual(encode(<<"\"">>, [escaped_strings]), [{string, <<"\\\"">>}, end_json])},
{"single quote escape", ?_assertEqual(encode(<<"'">>, [escaped_strings, single_quoted_strings]), [{string, <<"\\'">>}, end_json])},
{"no single quote escape", ?_assertEqual(encode(<<"'">>, [escaped_strings]), [{string, <<"'">>}, end_json])},
{"forward slash escape", ?_assertEqual(encode(<<"/">>, [escaped_strings, escaped_forward_slashes]), [{string, <<"\\/">>}, end_json])},
{"no forward slash escape", ?_assertEqual(encode(<<"/">>, [escaped_strings]), [{string, <<"/">>}, end_json])},
{"back slash escape", ?_assertEqual(encode(<<"\\">>, [escaped_strings]), [{string, <<"\\\\">>}, end_json])},
{"jsonp escape", ?_assertEqual(
encode(<<16#2028/utf8, 16#2029/utf8>>, [escaped_strings]),
[{string, <<"\\u2028\\u2029">>}, end_json]
)},
{"no jsonp escape", ?_assertEqual(
encode(<<16#2028/utf8, 16#2029/utf8>>, [escaped_strings, unescaped_jsonp]),
[{string, <<16#2028/utf8, 16#2029/utf8>>}, end_json]
)},
{"control escape", ?_assertEqual(encode(<<0>>, [escaped_strings]), [{string, <<"\\u0000">>}, end_json])},
{"dirty strings", ?_assertEqual(encode(<<"\n">>, [escaped_strings, dirty_strings]), [{string, <<"\n">>}, end_json])},
{"ignore bad escapes", ?_assertEqual(encode(<<"\\x25">>, [escaped_strings, ignored_bad_escapes]), [{string, <<"\\\\x25">>}, end_json])}
].
surrogates_test_() ->
[
{"surrogates - badjson",
?_assertEqual(check_bad(surrogates()), [])
?_assert(check_bad(surrogates()))
},
{"surrogates - replaced",
?_assertEqual(check_replaced(surrogates()), [])
?_assert(check_replaced(surrogates()))
}
].
@ -502,10 +915,25 @@ surrogates_test_() ->
good_characters_test_() ->
[
{"acceptable codepoints",
?_assertEqual(check_good(good()), [])
?_assert(check_good(good()))
},
{"acceptable codepoints - escaped_strings",
?_assert(check_good(good(), [escaped_strings]))
},
{"acceptable codepoints - replaced_bad_utf8",
?_assert(check_good(good(), [escaped_strings]))
},
{"acceptable codepoints - escaped_strings + replaced_bad_utf8",
?_assert(check_good(good(), [escaped_strings, replaced_bad_utf8]))
},
{"acceptable extended",
?_assertEqual(check_good(good_extended()), [])
?_assert(check_good(good_extended()))
},
{"acceptable extended - escaped_strings",
?_assert(check_good(good_extended(), [escaped_strings]))
},
{"acceptable extended - escaped_strings",
?_assert(check_good(good_extended(), [replaced_bad_utf8]))
}
].
@ -513,10 +941,10 @@ good_characters_test_() ->
reserved_test_() ->
[
{"reserved noncharacters - badjson",
?_assertEqual(check_bad(reserved_space()), [])
?_assert(check_bad(reserved_space()))
},
{"reserved noncharacters - replaced",
?_assertEqual(check_replaced(reserved_space()), [])
?_assert(check_replaced(reserved_space()))
}
].
@ -524,10 +952,10 @@ reserved_test_() ->
noncharacters_test_() ->
[
{"noncharacters - badjson",
?_assertEqual(check_bad(noncharacters()), [])
?_assert(check_bad(noncharacters()))
},
{"noncharacters - replaced",
?_assertEqual(check_replaced(noncharacters()), [])
?_assert(check_replaced(noncharacters()))
}
].
@ -535,31 +963,32 @@ noncharacters_test_() ->
extended_noncharacters_test_() ->
[
{"extended noncharacters - badjson",
?_assertEqual(check_bad(extended_noncharacters()), [])
?_assert(check_bad(extended_noncharacters()))
},
{"extended noncharacters - replaced",
?_assertEqual(check_replaced(extended_noncharacters()), [])
?_assert(check_replaced(extended_noncharacters()))
}
].
check_bad(List) ->
lists:dropwhile(fun({_, {error, badjson}}) -> true ; (_) -> false end,
[] == lists:dropwhile(fun({_, {error, badjson}}) -> true ; (_) -> false end,
check(List, [], [])
).
check_replaced(List) ->
lists:dropwhile(fun({_, [{string, <<16#fffd/utf8>>}|_]}) -> true
; (_) -> false
[] == lists:dropwhile(fun({_, [{string, <<16#fffd/utf8>>}|_]}) -> true ; (_) -> false
end,
check(List, [loose_unicode], [])
check(List, [replaced_bad_utf8], [])
).
check_good(List) ->
lists:dropwhile(fun({_, [{string, _}|_]}) -> true ; (_) -> false end,
check(List, [], [])
check_good(List) -> check_good(List, []).
check_good(List, Opts) ->
[] == lists:dropwhile(fun({_, [{string, _}|_]}) -> true ; (_) -> false end,
check(List, Opts, [])
).
@ -571,7 +1000,6 @@ check([H|T], Opts, Acc) ->
noncharacters() -> lists:seq(16#fffe, 16#ffff).
extended_noncharacters() ->
[16#1fffe, 16#1ffff, 16#2fffe, 16#2ffff]
++ [16#3fffe, 16#3ffff, 16#4fffe, 16#4ffff]
@ -582,16 +1010,12 @@ extended_noncharacters() ->
++ [16#dfffe, 16#dffff, 16#efffe, 16#effff]
++ [16#ffffe, 16#fffff, 16#10fffe, 16#10ffff].
surrogates() -> lists:seq(16#d800, 16#dfff).
reserved_space() -> lists:seq(16#fdd0, 16#fdef).
good() -> lists:seq(16#0000, 16#d7ff) ++ lists:seq(16#e000, 16#fdcf) ++ lists:seq(16#fdf0, 16#fffd).
good_extended() -> [16#10000, 16#20000, 16#30000, 16#40000, 16#50000,
16#60000, 16#70000, 16#80000, 16#90000, 16#a0000,
16#b0000, 16#c0000, 16#d0000, 16#e0000, 16#f0000

15
src/jsx_opts.hrl
View file

@ -1,11 +1,12 @@
-record(opts, {
loose_unicode = false,
escape_forward_slash = false,
explicit_end = false,
single_quotes = false,
no_jsonp_escapes = false,
replaced_bad_utf8 = false,
escaped_forward_slashes = false,
single_quoted_strings = false,
unescaped_jsonp = false,
comments = false,
json_escape = false,
escaped_strings = false,
dirty_strings = false,
ignore_bad_escapes = false
ignored_bad_escapes = false,
explicit_end = false,
pre_encode = false
}).

34
src/jsx_to_json.erl
View file

@ -39,13 +39,13 @@
-spec to_json(Source::any(), Opts::opts()) -> binary().
to_json(Source, Opts) when is_list(Opts) ->
(jsx:encoder(?MODULE, Opts, jsx_utils:extract_opts([json_escape] ++ Opts)))(Source).
(jsx:encoder(?MODULE, Opts, jsx_utils:extract_opts(Opts ++ [escaped_strings])))(Source).
-spec format(Source::binary(), Opts::opts()) -> binary().
format(Source, Opts) when is_binary(Source) andalso is_list(Opts) ->
(jsx:decoder(?MODULE, Opts, jsx_utils:extract_opts([json_escape] ++ Opts)))(Source).
(jsx:decoder(?MODULE, Opts, jsx_utils:extract_opts(Opts ++ [escaped_strings])))(Source).
parse_opts(Opts) -> parse_opts(Opts, #opts{}).
@ -142,7 +142,7 @@ encode(literal, Literal, _Opts) ->
encode(integer, Integer, _Opts) ->
erlang:integer_to_list(Integer);
encode(float, Float, _Opts) ->
nicedecimal:format(Float).
[Output] = io_lib:format("~p", [Float]), Output.
space(Opts) ->
@ -176,24 +176,12 @@ indent_or_space(Opts) ->
-ifdef(TEST).
-include_lib("eunit/include/eunit.hrl").
setup_nicedecimal_meck(Return) ->
ok = meck:new(nicedecimal),
ok = meck:expect(nicedecimal, format, fun(1.23) -> Return end).
teardown_nicedecimal_meck(_) ->
?assert(meck:validate(nicedecimal)),
ok = meck:unload(nicedecimal).
basic_format_test_() ->
[
{"empty object", ?_assertEqual(format(<<"{}">>, []), <<"{}">>)},
{"empty array", ?_assertEqual(format(<<"[]">>, []), <<"[]">>)},
{"naked integer", ?_assertEqual(format(<<"123">>, []), <<"123">>)},
{foreach,
fun() -> setup_nicedecimal_meck(<<"1.23">>) end,
fun(R) -> teardown_nicedecimal_meck(R) end,
[{"naked float", ?_assertEqual(format(<<"1.23">>, []), <<"1.23">>)}]
},
{"naked float", ?_assertEqual(format(<<"1.23">>, []), <<"1.23">>)},
{"naked string", ?_assertEqual(format(<<"\"hi\"">>, []), <<"\"hi\"">>)},
{"naked string with control character", ?_assertEqual(
format(<<"\"hi\\n\"">>, []), <<"\"hi\\n\"">>
@ -238,11 +226,7 @@ basic_to_json_test_() ->
{"empty object", ?_assertEqual(to_json([{}], []), <<"{}">>)},
{"empty array", ?_assertEqual(to_json([], []), <<"[]">>)},
{"naked integer", ?_assertEqual(to_json(123, []), <<"123">>)},
{foreach,
fun() -> setup_nicedecimal_meck(<<"1.23">>) end,
fun(R) -> teardown_nicedecimal_meck(R) end,
[{"naked float", ?_assertEqual(to_json(1.23, []) , <<"1.23">>)}]
},
{"naked float", ?_assertEqual(to_json(1.23, []) , <<"1.23">>)},
{"naked string", ?_assertEqual(to_json(<<"hi">>, []), <<"\"hi\"">>)},
{"naked string with control character", ?_assertEqual(
to_json(<<"hi\n">>, []), <<"\"hi\\n\"">>
@ -316,14 +300,10 @@ opts_test_() ->
format(<<"{\"a\":true,\"b\":true,\"c\":true}">>, [{space, 2}]),
<<"{\"a\": true, \"b\": true, \"c\": true}">>
)},
{foreach,
fun() -> setup_nicedecimal_meck(<<"1.23">>) end,
fun(R) -> teardown_nicedecimal_meck(R) end,
[{"array indent", ?_assertEqual(
{"array indent", ?_assertEqual(
format(<<"[1.23, 1.23, 1.23]">>, [{indent, 2}]),
<<"[\n 1.23,\n 1.23,\n 1.23\n]">>
)}]
},
)},
{"object indent", ?_assertEqual(
format(<<"{\"a\":true,\"b\":true,\"c\":true}">>, [{indent, 2}]),
<<"{\n \"a\":true,\n \"b\":true,\n \"c\":true\n}">>

71
src/jsx_to_term.erl
View file

@ -28,7 +28,8 @@
-record(opts, {
labels = binary
labels = binary,
post_decode = false
}).
-type opts() :: list().
@ -49,7 +50,6 @@ to_term(Source, Opts) when is_list(Opts) ->
(jsx:decoder(?MODULE, Opts, jsx_utils:extract_opts(Opts)))(Source).
parse_opts(Opts) -> parse_opts(Opts, #opts{}).
parse_opts([{labels, Val}|Rest], Opts)
@ -57,42 +57,43 @@ parse_opts([{labels, Val}|Rest], Opts)
parse_opts(Rest, Opts#opts{labels = Val});
parse_opts([labels|Rest], Opts) ->
parse_opts(Rest, Opts#opts{labels = binary});
parse_opts([{post_decode, F}|Rest], Opts=#opts{post_decode=false}) when is_function(F, 1) ->
parse_opts(Rest, Opts#opts{post_decode=F});
parse_opts([{post_decode, _}|_] = Options, Opts) ->
erlang:error(badarg, [Options, Opts]);
parse_opts([_|Rest], Opts) ->
parse_opts(Rest, Opts);
parse_opts([], Opts) ->
Opts.
init(Opts) -> {[[]], parse_opts(Opts)}.
handle_event(end_json, {[[Terms]], _Opts}) -> Terms;
handle_event(start_object, {Terms, Opts}) -> {[[]|Terms], Opts};
handle_event(end_object, {[[], {key, Key}, Last|Terms], Opts}) ->
{[[{Key, [{}]}] ++ Last] ++ Terms, Opts};
{[[{Key, post_decode([{}], Opts)}] ++ Last] ++ Terms, Opts};
handle_event(end_object, {[Object, {key, Key}, Last|Terms], Opts}) ->
{[[{Key, lists:reverse(Object)}] ++ Last] ++ Terms, Opts};
{[[{Key, post_decode(lists:reverse(Object), Opts)}] ++ Last] ++ Terms, Opts};
handle_event(end_object, {[[], Last|Terms], Opts}) ->
{[[[{}]] ++ Last] ++ Terms, Opts};
{[[post_decode([{}], Opts)] ++ Last] ++ Terms, Opts};
handle_event(end_object, {[Object, Last|Terms], Opts}) ->
{[[lists:reverse(Object)] ++ Last] ++ Terms, Opts};
{[[post_decode(lists:reverse(Object), Opts)] ++ Last] ++ Terms, Opts};
handle_event(start_array, {Terms, Opts}) -> {[[]|Terms], Opts};
handle_event(end_array, {[List, {key, Key}, Last|Terms], Opts}) ->
{[[{Key, lists:reverse(List)}] ++ Last] ++ Terms, Opts};
{[[{Key, post_decode(lists:reverse(List), Opts)}] ++ Last] ++ Terms, Opts};
handle_event(end_array, {[Current, Last|Terms], Opts}) ->
{[[lists:reverse(Current)] ++ Last] ++ Terms, Opts};
{[[post_decode(lists:reverse(Current), Opts)] ++ Last] ++ Terms, Opts};
handle_event({key, Key}, {Terms, Opts}) -> {[{key, format_key(Key, Opts)}] ++ Terms, Opts};
handle_event({_, Event}, {[{key, Key}, Last|Terms], Opts}) ->
{[[{Key, Event}] ++ Last] ++ Terms, Opts};
{[[{Key, post_decode(Event, Opts)}] ++ Last] ++ Terms, Opts};
handle_event({_, Event}, {[Last|Terms], Opts}) ->
{[[Event] ++ Last] ++ Terms, Opts}.
{[[post_decode(Event, Opts)] ++ Last] ++ Terms, Opts}.
format_key(Key, Opts) ->
@ -103,6 +104,9 @@ format_key(Key, Opts) ->
end.
post_decode(Value, #opts{post_decode=false}) -> Value;
post_decode(Value, Opts) -> (Opts#opts.post_decode)(Value).
%% eunit tests
@ -182,4 +186,45 @@ naked_test_() ->
{"naked string", ?_assertEqual(to_term(<<"\"string\"">>, []), <<"string">>)}
].
post_decoders_test_() ->
JSON = <<"{\"object\": {
\"literals\": [true, false, null],
\"strings\": [\"foo\", \"bar\", \"baz\"],
\"numbers\": [1, 1.0, 1e0]
}}">>,
[
{"no post_decode", ?_assertEqual(
to_term(JSON, []),
[{<<"object">>, [
{<<"literals">>, [true, false, null]},
{<<"strings">>, [<<"foo">>, <<"bar">>, <<"baz">>]},
{<<"numbers">>, [1, 1.0, 1.0]}
]}]
)},
{"replace arrays with empty arrays", ?_assertEqual(
to_term(JSON, [{post_decode, fun([T|_] = V) when is_tuple(T) -> V; (V) when is_list(V) -> []; (V) -> V end}]),
[{<<"object">>, [{<<"literals">>, []}, {<<"strings">>, []}, {<<"numbers">>, []}]}]
)},
{"replace objects with empty objects", ?_assertEqual(
to_term(JSON, [{post_decode, fun(V) when is_list(V) -> [{}]; (V) -> V end}]),
[{}]
)},
{"replace all non-list values with false", ?_assertEqual(
to_term(JSON, [{post_decode, fun(V) when is_list(V) -> V; (_) -> false end}]),
[{<<"object">>, [
{<<"literals">>, [false, false, false]},
{<<"strings">>, [false, false, false]},
{<<"numbers">>, [false, false, false]}
]}]
)},
{"atoms_to_strings", ?_assertEqual(
to_term(JSON, [{post_decode, fun(V) when is_atom(V) -> unicode:characters_to_binary(atom_to_list(V)); (V) -> V end}]),
[{<<"object">>, [
{<<"literals">>, [<<"true">>, <<"false">>, <<"null">>]},
{<<"strings">>, [<<"foo">>, <<"bar">>, <<"baz">>]},
{<<"numbers">>, [1, 1.0, 1.0]}
]}]
)}
].
-endif.

376
src/jsx_utils.erl
View file

@ -25,59 +25,90 @@
-export([parse_opts/1]).
-export([extract_opts/1]).
-export([json_escape/2]).
-export([json_escape_sequence/1]).
-include("jsx_opts.hrl").
%% parsing of jsx opts
parse_opts(Opts) ->
parse_opts(Opts, #opts{}).
parse_opts([], Opts) ->
Opts;
parse_opts([loose_unicode|Rest], Opts) ->
parse_opts(Rest, Opts#opts{loose_unicode=true});
parse_opts([escape_forward_slash|Rest], Opts) ->
parse_opts(Rest, Opts#opts{escape_forward_slash=true});
parse_opts([replaced_bad_utf8|Rest], Opts) ->
parse_opts(Rest, Opts#opts{replaced_bad_utf8=true});
parse_opts([escaped_forward_slashes|Rest], Opts) ->
parse_opts(Rest, Opts#opts{escaped_forward_slashes=true});
parse_opts([explicit_end|Rest], Opts) ->
parse_opts(Rest, Opts#opts{explicit_end=true});
parse_opts([single_quotes|Rest], Opts) ->
parse_opts(Rest, Opts#opts{single_quotes=true});
parse_opts([no_jsonp_escapes|Rest], Opts) ->
parse_opts(Rest, Opts#opts{no_jsonp_escapes=true});
parse_opts([single_quoted_strings|Rest], Opts) ->
parse_opts(Rest, Opts#opts{single_quoted_strings=true});
parse_opts([unescaped_jsonp|Rest], Opts) ->
parse_opts(Rest, Opts#opts{unescaped_jsonp=true});
parse_opts([comments|Rest], Opts) ->
parse_opts(Rest, Opts#opts{comments=true});
parse_opts([json_escape|Rest], Opts) ->
parse_opts(Rest, Opts#opts{json_escape=true});
parse_opts([escaped_strings|Rest], Opts) ->
parse_opts(Rest, Opts#opts{escaped_strings=true});
parse_opts([dirty_strings|Rest], Opts) ->
parse_opts(Rest, Opts#opts{dirty_strings=true});
parse_opts([ignore_bad_escapes|Rest], Opts) ->
parse_opts(Rest, Opts#opts{ignore_bad_escapes=true});
parse_opts([ignored_bad_escapes|Rest], Opts) ->
parse_opts(Rest, Opts#opts{ignored_bad_escapes=true});
parse_opts([relax|Rest], Opts) ->
parse_opts(Rest, Opts#opts{
loose_unicode = true,
single_quotes = true,
replaced_bad_utf8 = true,
single_quoted_strings = true,
comments = true,
ignore_bad_escapes = true
ignored_bad_escapes = true
});
parse_opts(_, _) ->
{error, badarg}.
parse_opts([{pre_encode, Encoder}|Rest] = Options, Opts) when is_function(Encoder, 1) ->
case Opts#opts.pre_encode of
false -> parse_opts(Rest, Opts#opts{pre_encode=Encoder})
; _ -> erlang:error(badarg, [Options, Opts])
end;
%% deprecated flags
parse_opts([{pre_encoder, Encoder}|Rest] = Options, Opts) when is_function(Encoder, 1) ->
case Opts#opts.pre_encode of
false -> parse_opts(Rest, Opts#opts{pre_encode=Encoder})
; _ -> erlang:error(badarg, [Options, Opts])
end;
parse_opts([loose_unicode|Rest], Opts) ->
parse_opts(Rest, Opts#opts{replaced_bad_utf8=true});
parse_opts([escape_forward_slash|Rest], Opts) ->
parse_opts(Rest, Opts#opts{escaped_forward_slashes=true});
parse_opts([single_quotes|Rest], Opts) ->
parse_opts(Rest, Opts#opts{single_quoted_strings=true});
parse_opts([no_jsonp_escapes|Rest], Opts) ->
parse_opts(Rest, Opts#opts{unescaped_jsonp=true});
parse_opts([json_escape|Rest], Opts) ->
parse_opts(Rest, Opts#opts{escaped_strings=true});
parse_opts([ignore_bad_escapes|Rest], Opts) ->
parse_opts(Rest, Opts#opts{ignored_bad_escapes=true});
parse_opts(Options, Opts) ->
erlang:error(badarg, [Options, Opts]).
valid_flags() ->
[
loose_unicode,
escape_forward_slash,
explicit_end,
single_quotes,
no_jsonp_escapes,
replaced_bad_utf8,
escaped_forward_slashes,
single_quoted_strings,
unescaped_jsonp,
comments,
json_escape,
escaped_strings,
dirty_strings,
ignore_bad_escapes,
relax
ignored_bad_escapes,
explicit_end,
relax,
pre_encode,
%% deprecated flags
pre_encoder, %% pre_encode
loose_unicode, %% replaced_bad_utf8
escape_forward_slash, %% escaped_forward_slashes
single_quotes, %% single_quotes_strings
no_jsonp_escapes, %% unescaped_jsonp
json_escape, %% escaped_strings
ignore_bad_escapes %% ignored_bad_escapes
].
@ -97,231 +128,10 @@ extract_parser_opts([K|Rest], Acc) ->
end.
%% json string escaping, for utf8 binaries. escape the json control sequences to
%% their json equivalent, escape other control characters to \uXXXX sequences,
%% everything else should be a legal json string component
json_escape(String, Opts) when is_binary(String) ->
case Opts#opts.dirty_strings of
true -> String
; false -> json_escape(String, Opts, 0, size(String))
end.
-define(control_character(X),
<<H:L/binary, X, T/binary>> ->
json_escape(
<<H/binary, (unicode:characters_to_binary(json_escape_sequence(X)))/binary, T/binary>>,
Opts,
L + 6,
Len + 5
)
).
json_escape(Str, Opts, L, Len) when L < Len ->
case Str of
?control_character(0);
?control_character(1);
?control_character(2);
?control_character(3);
?control_character(4);
?control_character(5);
?control_character(6);
?control_character(7);
<<H:L/binary, $\b, T/binary>> -> json_escape(<<H/binary, $\\, $b, T/binary>>, Opts, L + 2, Len + 1);
<<H:L/binary, $\t, T/binary>> -> json_escape(<<H/binary, $\\, $t, T/binary>>, Opts, L + 2, Len + 1);
<<H:L/binary, $\n, T/binary>> -> json_escape(<<H/binary, $\\, $n, T/binary>>, Opts, L + 2, Len + 1);
?control_character(11);
<<H:L/binary, $\f, T/binary>> -> json_escape(<<H/binary, $\\, $f, T/binary>>, Opts, L + 2, Len + 1);
<<H:L/binary, $\r, T/binary>> -> json_escape(<<H/binary, $\\, $r, T/binary>>, Opts, L + 2, Len + 1);
?control_character(14);
?control_character(15);
?control_character(16);
?control_character(17);
?control_character(18);
?control_character(19);
?control_character(20);
?control_character(21);
?control_character(22);
?control_character(23);
?control_character(24);
?control_character(25);
?control_character(26);
?control_character(27);
?control_character(28);
?control_character(29);
?control_character(30);
?control_character(31);
<<_:L/binary, 32, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 33, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<H:L/binary, $\", T/binary>> -> json_escape(<<H/binary, $\\, $", T/binary>>, Opts, L + 2, Len + 1);
<<_:L/binary, 35, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 36, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 37, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 38, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 39, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 40, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 41, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 42, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 43, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 44, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 45, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 46, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<H:L/binary, $/, T/binary>> ->
case Opts#opts.escape_forward_slash of
true ->
json_escape(<<H/binary, $\\, $/, T/binary>>, Opts, L + 2, Len + 1);
false ->
json_escape(<<H/binary, $/, T/binary>>, Opts, L + 1, Len)
end;
<<_:L/binary, 48, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 49, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 50, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 51, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 52, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 53, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 54, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 55, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 56, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 57, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 58, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 59, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 60, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 61, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 62, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 63, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 64, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 65, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 66, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 67, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 68, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 69, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 70, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 71, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 72, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 73, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 74, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 75, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 76, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 77, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 78, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 79, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 80, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 81, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 82, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 83, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 84, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 85, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 86, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 87, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 88, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 89, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 90, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 91, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<H:L/binary, $\\, T/binary>> -> json_escape(<<H/binary, $\\, $\\, T/binary>>, Opts, L + 2, Len + 1);
<<_:L/binary, 93, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 94, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 95, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 96, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 97, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 98, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 99, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 100, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 101, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 102, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 103, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 104, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 105, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 106, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 107, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 108, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 109, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 110, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 111, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 112, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 113, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 114, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 115, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 116, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 117, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 118, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 119, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 120, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 121, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 122, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 123, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 124, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 125, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 126, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, 127, _/binary>> -> json_escape(Str, Opts, L + 1, Len);
<<H:L/binary, 16#2028/utf8, T/binary>> ->
case Opts#opts.no_jsonp_escapes of
true ->
json_escape(<<H/binary, 16#2028/utf8, T/binary>>, Opts, L + 3, Len);
false ->
B = unicode:characters_to_binary(json_escape_sequence(16#2028)),
json_escape(<<H/binary, B/binary, T/binary>>, Opts, L + 6, Len + 3)
end;
<<H:L/binary, 16#2029/utf8, T/binary>> ->
case Opts#opts.no_jsonp_escapes of
true ->
json_escape(<<H/binary, 16#2029/utf8, T/binary>>, Opts, L + 3, Len);
false ->
B = unicode:characters_to_binary(json_escape_sequence(16#2029)),
json_escape(<<H/binary, B/binary, T/binary>>, Opts, L + 6, Len + 3)
end;
<<_:L/binary, X/utf8, _/binary>> when X < 16#0080 ->
json_escape(Str, Opts, L + 1, Len);
<<_:L/binary, X/utf8, _/binary>> when X < 16#0800 ->
json_escape(Str, Opts, L + 2, Len);
<<_:L/binary, X/utf8, _/binary>> when X < 16#dcff ->
json_escape(Str, Opts, L + 3, Len);
<<_:L/binary, X/utf8, _/binary>> when X > 16#dfff, X < 16#fdd0 ->
json_escape(Str, Opts, L + 3, Len);
<<_:L/binary, X/utf8, _/binary>> when X > 16#fdef, X < 16#fffe ->
json_escape(Str, Opts, L + 3, Len);
<<_:L/binary, X/utf8, _/binary>> when X >= 16#10000, X < 16#1fffe ->
json_escape(Str, Opts, L + 4, Len);
<<_:L/binary, X/utf8, _/binary>> when X >= 16#20000, X < 16#2fffe ->
json_escape(Str, Opts, L + 4, Len);
<<_:L/binary, X/utf8, _/binary>> when X >= 16#30000, X < 16#3fffe ->
json_escape(Str, Opts, L + 4, Len);
<<_:L/binary, X/utf8, _/binary>> when X >= 16#40000, X < 16#4fffe ->
json_escape(Str, Opts, L + 4, Len);
<<_:L/binary, X/utf8, _/binary>> when X >= 16#50000, X < 16#5fffe ->
json_escape(Str, Opts, L + 4, Len);
<<_:L/binary, X/utf8, _/binary>> when X >= 16#60000, X < 16#6fffe ->
json_escape(Str, Opts, L + 4, Len);
<<_:L/binary, X/utf8, _/binary>> when X >= 16#70000, X < 16#7fffe ->
json_escape(Str, Opts, L + 4, Len);
<<_:L/binary, X/utf8, _/binary>> when X >= 16#80000, X < 16#8fffe ->
json_escape(Str, Opts, L + 4, Len);
<<_:L/binary, X/utf8, _/binary>> when X >= 16#90000, X < 16#9fffe ->
json_escape(Str, Opts, L + 4, Len);
<<_:L/binary, X/utf8, _/binary>> when X >= 16#a0000, X < 16#afffe ->
json_escape(Str, Opts, L + 4, Len);
<<_:L/binary, X/utf8, _/binary>> when X >= 16#b0000, X < 16#bfffe ->
json_escape(Str, Opts, L + 4, Len);
<<_:L/binary, X/utf8, _/binary>> when X >= 16#c0000, X < 16#cfffe ->
json_escape(Str, Opts, L + 4, Len);
<<_:L/binary, X/utf8, _/binary>> when X >= 16#d0000, X < 16#dfffe ->
json_escape(Str, Opts, L + 4, Len);
<<_:L/binary, X/utf8, _/binary>> when X >= 16#e0000, X < 16#efffe ->
json_escape(Str, Opts, L + 4, Len);
<<_:L/binary, X/utf8, _/binary>> when X >= 16#f0000, X < 16#ffffe ->
json_escape(Str, Opts, L + 4, Len);
<<_:L/binary, X/utf8, _/binary>> when X >= 16#100000, X < 16#10fffe ->
json_escape(Str, Opts, L + 4, Len);
_ -> erlang:error(badarg, [Str, Opts])
end;
json_escape(Str, _, L, Len) when L =:= Len ->
Str.
%% convert a codepoint to it's \uXXXX equiv.
json_escape_sequence(X) ->
<<A:4, B:4, C:4, D:4>> = <<X:16>>,
unicode:characters_to_binary([$\\, $u, (to_hex(A)), (to_hex(B)), (to_hex(C)), (to_hex(D))]).
[$\\, $u, (to_hex(A)), (to_hex(B)), (to_hex(C)), (to_hex(D))].
to_hex(10) -> $a;
@ -338,70 +148,36 @@ to_hex(X) -> X + 48. %% ascii "1" is [49], "2" is [50], etc...
-include_lib("eunit/include/eunit.hrl").
binary_escape_test_() ->
json_escape_sequence_test_() ->
[
{"json string escaping",
?_assertEqual(
json_escape(<<"\"\\\b\f\n\r\t">>, #opts{}),
<<"\\\"\\\\\\b\\f\\n\\r\\t">>
)
},
{"json string hex escape",
?_assertEqual(
json_escape(<<0, 1, 2, 3, 11, 26, 30, 31>>, #opts{}),
<<"\\u0000\\u0001\\u0002\\u0003\\u000b\\u001a\\u001e\\u001f">>
)
},
{"jsonp protection",
?_assertEqual(
json_escape(<<226, 128, 168, 226, 128, 169>>, #opts{}),
<<"\\u2028\\u2029">>
)
},
{"no jsonp escapes",
?_assertEqual(
json_escape(<<226, 128, 168, 226, 128, 169>>, #opts{no_jsonp_escapes=true}),
<<226, 128, 168, 226, 128, 169>>
)
},
{"microsoft i hate your date format",
?_assertEqual(
json_escape(<<"/Date(1303502009425)/">>, #opts{escape_forward_slash=true}),
<<"\\/Date(1303502009425)\\/">>
)
},
{"dirty strings",
?_assertEqual(
json_escape(<<"\\x25\\uffff">>, #opts{dirty_strings=true}),
<<"\\x25\\uffff">>
)
}
{"json escape sequence test - 16#0000", ?_assertEqual(json_escape_sequence(16#0000), "\\u0000")},
{"json escape sequence test - 16#abc", ?_assertEqual(json_escape_sequence(16#abc), "\\u0abc")},
{"json escape sequence test - 16#def", ?_assertEqual(json_escape_sequence(16#def), "\\u0def")}
].
opts_test_() ->
[
{"all flags",
?_assertEqual(
parse_opts([
loose_unicode,
escape_forward_slash,
replaced_bad_utf8,
escaped_forward_slashes,
explicit_end,
single_quotes,
no_jsonp_escapes,
single_quoted_strings,
unescaped_jsonp,
comments,
dirty_strings,
ignore_bad_escapes
ignored_bad_escapes
]),
#opts{
loose_unicode=true,
escape_forward_slash=true,
replaced_bad_utf8=true,
escaped_forward_slashes=true,
explicit_end=true,
single_quotes=true,
no_jsonp_escapes=true,
single_quoted_strings=true,
unescaped_jsonp=true,
comments=true,
dirty_strings=true,
ignore_bad_escapes=true
ignored_bad_escapes=true
}
)
},
@ -409,10 +185,10 @@ opts_test_() ->
?_assertEqual(
parse_opts([relax]),
#opts{
loose_unicode=true,
single_quotes=true,
replaced_bad_utf8=true,
single_quoted_strings=true,
comments=true,
ignore_bad_escapes=true
ignored_bad_escapes=true
}
)
}