Lexical Analysis: Hard Parts

How does one do lexical analysis (tokenization) of Bash/Perl style string, where insertion in the middle is possible?
Say we have
print "Now is {get "time of day"} of {get "current date"}..."
print 'have a nice day!'
And {…} inserts value in the middle of a string.

How Lexer would know which double-quote closes the string and which is part of the string?

Python uses format-like routines, because parsing such strings is hard.

HMMM LET'S SEE WE CREATE A STATE MACHINE THAT GOES THROUGH EACH CHARACTER IN THE FUQIN STRING AND WHEN WE ENCOUNTER A FUQIN QUOTE, WE INCREMENT A VARIABLE CALLED QUOTE_COUNT OR SOME SHIT LIKE THAT.  AND...

WAIT, I DON'T FUCKING KNOW WHAT I'M DOING
FUCK.

No, that is a good way to do it >>9-sama.

>>9
You don't understand the question.

Use Perl 6 grammar.

ok here Is what to does

loop from begin of str til u hit quote
when u hit quote , stop loop, then 2nd loop from end of str BACKWARD til u hit END quote.

recurse for the winnings

>>9
>WE INCREMENT A VARIABLE CALLED QUOTE
what if there are uneven number of quotes?

>>14
That would be a syntax error.

>>13
loop from end of str BACKWARD
see http://en.wikipedia.org/wiki/Look-ahead_%28backtracking%29

>>15
But it isnt! For example,
http://perl.about.com/od/perltutorials/qt/perlheredoc.htm
can contain arbitrary characters, which is very useful for rapid prototyping.

>>17
Obviously he intended that to be applied only in the particular case...