Improve Ripper compatibiliy layer and emit :on_embexpr_end for unmatched closing brace

[andrykonchin]

Changes:

• emit :on_embexpr_end for the first unmatched }

Rationale

Ripper contains a scanner quirk that tokenizes the first unmatched closing curly brace ( }) as :on_embexpr_end rather than :on_rbrace. Because Prism did not replicate this behavior, Haml attribute parsing (which relies on this quirk to balance braces) failed with an Haml::SyntaxError: Unbalanced brackets error.

Right now the Haml and Sinatra tests fail on TruffleRuby on CI. This patch fixes the specs (checked locally).

cc @eregon

Examples (before the fix)

Ripper:

Ripper.lex("a(:class => :header}")
=>
[[[1, 0], :on_ident, "a", CMDARG],
 [[1, 1], :on_lparen, "(", BEG|LABEL],
 [[1, 2], :on_symbeg, ":", FNAME],
 [[1, 3], :on_kw, "class", ENDFN],
 [[1, 8], :on_sp, " ", END],
 [[1, 9], :on_op, "=>", BEG],
 [[1, 11], :on_sp, " ", BEG],
 [[1, 12], :on_symbeg, ":", FNAME],
 [[1, 13], :on_ident, "header", ENDFN],
 [[1, 19], :on_embexpr_end, "}", END]]

Prism:

Prism.lex_compat("a(:class => :header}")
=>
#<Prism::LexCompat::Result:0x00000001280bc948
 @comments=[],
 @data_loc=nil,
 @errors=
  [#<Prism::ParseError @type=:argument_term_paren @message="unexpected '}'; expected a `)` to close the arguments" @location=#<Prism::Location @start_offset=19 @length=1 start_line=1> @level=:syntax>,
   #<Prism::ParseError @type=:expect_eol_after_statement @message="unexpected '}', expecting end-of-input" @location=#<Prism::Location @start_offset=19 @length=1 start_line=1> @level=:syntax>,
   #<Prism::ParseError @type=:unexpected_token_ignore @message="unexpected '}', ignoring it" @location=#<Prism::Location @start_offset=19 @length=1 start_line=1> @level=:syntax>],
 @magic_comments=[],
 @source=#<Prism::ASCIISource:0x00000001280d9d18 @offsets=[0], @source="a(:class => :header}", @start_line=1>,
 @value=
  [[[1, 0], :on_ident, "a", CMDARG],
   [[1, 1], :on_lparen, "(", BEG|LABEL],
   [[1, 2], :on_symbeg, ":", FNAME],
   [[1, 3], :on_kw, "class", ENDFN],
   [[1, 8], :on_sp, " ", ENDFN],
   [[1, 9], :on_op, "=>", BEG],
   [[1, 11], :on_sp, " ", BEG],
   [[1, 12], :on_symbeg, ":", FNAME],
   [[1, 13], :on_ident, "header", ENDFN],
   [[1, 19], :on_rbrace, "}", END]],
 @warnings=[]>

The Haml use case

When rendering a Haml tag with attributes (e.g., %h1{:class => :header} Hello World), Haml needs to parse and extract the attributes hash. It does this by temporarily replacing the leading { with a dummy method call prefix a(, transforming it into a(:class => :header}.

It then tokenizes this string using Ripper.lex to balance the brackets and extract the attributes hash. Since the opening { was replaced with (, the closing } is unmatched.

https://github.com/haml/haml/blob/08808371860b7289af6ef23f328ae42f81a19afc/lib/haml/parser.rb#L686-L692

[Earlopain]

I'm not 100% convinded we should do this. assert_ripper_lex("} } { {") for example fails because whatever ripper does returns the first { as tlambeg, but only if brackets were initially unbalanced.

More generally, relying on the shape of syntax invalid tokens seems like a horrible idea and subject to change without notice. Haml transforms syntax invalid code into slightly different syntax invalid code and when this got reported in #4127 I suggested to simply stop doing that.

Would haml/haml#1210 work for you (I suppose you may have already seen this)?

[andrykonchin]

Ah, I completely overlooked the mentioned PR haml/haml#1210. At first glance the changes look good enough. Thank you!

Will restore the PR if haml/haml#1210 is rejected.