"Fossies" - the Fresh Open Source Software Archive

Member "AutoHotkey_L-1.1.33.09/source/script_expression.cpp" (8 May 2021, 163507 Bytes) of package /windows/misc/AutoHotkey_L-1.1.33.09.zip:


As a special service "Fossies" has tried to format the requested source page into HTML format using (guessed) C and C++ source code syntax highlighting (style: standard) with prefixed line numbers and code folding option. Alternatively you can here view or download the uninterpreted source code file. For more information about "script_expression.cpp" see the Fossies "Dox" file reference documentation.

    1 /*
    2 AutoHotkey
    3 
    4 Copyright 2003-2009 Chris Mallett (support@autohotkey.com)
    5 
    6 This program is free software; you can redistribute it and/or
    7 modify it under the terms of the GNU General Public License
    8 as published by the Free Software Foundation; either version 2
    9 of the License, or (at your option) any later version.
   10 
   11 This program is distributed in the hope that it will be useful,
   12 but WITHOUT ANY WARRANTY; without even the implied warranty of
   13 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
   14 GNU General Public License for more details.
   15 */
   16 
   17 //////////////////////////////////////////////////////////////////////////////////////
   18 // v1.0.40.02: This is now a separate file to allow its compiler optimization settings
   19 // to be set independently of those of the other modules.  In one benchmark, this
   20 // improved performance of expressions and function calls by 9% (that is, when the
   21 // other modules are set to "minimize size" such as for the AutoHotkeySC.bin file).
   22 // This gain in performance is at the cost of a 1.5 KB increase in the size of the
   23 // compressed code, which seems well worth it given how often expressions and
   24 // function-calls are used (such as in loops).
   25 //
   26 // ExpandArgs() and related functions were also put into this file because that
   27 // further improves performance across the board -- even for AutoHotkey.exe despite
   28 // the fact that the only thing that changed for it was the module move, not the
   29 // compiler settings.  Apparently, the butterfly effect can cause even minor
   30 // modifications to impact the overall performance of the generated code by as much as
   31 // 7%.  However, this might have more to do with cache hits and misses in the CPU than
   32 // with the nature of the code produced by the compiler.
   33 // UPDATE 10/18/2006: There's not much difference anymore -- in fact, using min size
   34 // for everything makes compiled scripts slightly faster in basic benchmarks, probably
   35 // due to the recent addition of the linker optimization that physically orders
   36 // functions in a better order inside the EXE.  Therefore, script_expression.cpp no
   37 // longer has a separate "favor fast code" option.
   38 //////////////////////////////////////////////////////////////////////////////////////
   39 
   40 #include "stdafx.h" // pre-compiled headers
   41 #include "script.h"
   42 #include "script_object.h"
   43 #include "globaldata.h" // for a lot of things
   44 #include "qmath.h" // For ExpandExpression()
   45 
   46 // __forceinline: Decided against it for this function because although it's only called by one caller,
   47 // testing shows that it wastes stack space (room for its automatic variables would be unconditionally 
   48 // reserved in the stack of its caller).  Also, the performance benefit of inlining this is too slight.
   49 // Here's a simple way to verify wasted stack space in a caller that calls an inlined function:
   50 //    DWORD stack
   51 //    _asm mov stack, esp
   52 //    MsgBox(stack);
   53 LPTSTR Line::ExpandExpression(int aArgIndex, ResultType &aResult, ExprTokenType *aResultToken
   54         , LPTSTR &aTarget, LPTSTR &aDerefBuf, size_t &aDerefBufSize, LPTSTR aArgDeref[], size_t aExtraSize)
   55 // Caller should ignore aResult unless this function returns NULL.
   56 // Returns a pointer to this expression's result, which can be one of the following:
   57 // 1) NULL, in which case aResult will be either FAIL or EARLY_EXIT to indicate the means by which the current
   58 //    quasi-thread was terminated as a result of a function call.
   59 // 2) The constant empty string (""), in which case we do not alter aTarget for our caller.
   60 // 3) Some persistent location not in aDerefBuf, namely the mContents of a variable or a literal string/number,
   61 //    such as a function-call that returns "abc", 123, or a variable.
   62 // 4) At position aTarget inside aDerefBuf (note that aDerefBuf might have been reallocated by us).
   63 // aTarget is left unchanged except in case #4, in which case aTarget has been adjusted to the position after our
   64 // result-string's terminator.  In addition, in case #4, aDerefBuf, aDerefBufSize, and aArgDeref[] have been adjusted
   65 // for our caller if aDerefBuf was too small and needed to be enlarged.
   66 //
   67 // Thanks to Joost Mulders for providing the expression evaluation code upon which this function is based.
   68 {
   69     LPTSTR target = aTarget; // "target" is used to track our usage (current position) within the aTarget buffer.
   70 
   71     // The following must be defined early so that mem_count is initialized and the array is guaranteed to be
   72     // "in scope" in case of early "goto" (goto substantially boosts performance and reduces code size here).
   73     #define MAX_EXPR_MEM_ITEMS 200 // v1.0.47.01: Raised from 100 because a line consisting entirely of concat operators can exceed it.  However, there's probably not much point to going much above MAX_TOKENS/2 because then it would reach the MAX_TOKENS limit first.
   74     LPTSTR mem[MAX_EXPR_MEM_ITEMS]; // No init necessary.  In most cases, it will never be used.
   75     int mem_count = 0; // The actual number of items in use in the above array.
   76     LPTSTR result_to_return = _T(""); // By contrast, NULL is used to tell the caller to abort the current thread.  That isn't done for normal syntax errors, just critical conditions such as out-of-memory.
   77     Var *output_var = (mActionType == ACT_ASSIGNEXPR) ? OUTPUT_VAR : NULL; // Resolve early because it's similar in usage/scope to the above.  Plus MUST be resolved prior to calling any script-functions since they could change the values in sArgVar[].
   78 
   79     ExprTokenType *stack[MAX_TOKENS];
   80     int stack_count = 0, high_water_mark = 0; // L31: high_water_mark is used to simplify object management.
   81     ExprTokenType *&postfix = mArg[aArgIndex].postfix;
   82 
   83     DerefType *deref;
   84     LPTSTR cp;
   85 
   86     ///////////////////////////////
   87     // EVALUATE POSTFIX EXPRESSION
   88     ///////////////////////////////
   89     int i, actual_param_count, delta;
   90     SymbolType right_is_number, left_is_number, result_symbol;
   91     double right_double, left_double;
   92     __int64 right_int64, left_int64;
   93     LPTSTR right_string, left_string;
   94     size_t right_length, left_length;
   95     TCHAR left_buf[MAX_NUMBER_SIZE];  // BIF_OnMessage and SYM_DYNAMIC rely on this one being large enough to hold MAX_VAR_NAME_LENGTH.
   96     TCHAR right_buf[MAX_NUMBER_SIZE]; // Only needed for holding numbers
   97     LPTSTR result; // "result" is used for return values and also the final result.
   98     VarSizeType result_length;
   99     size_t result_size, alloca_usage = 0; // v1.0.45: Track amount of alloca mem to avoid stress on stack from extreme expressions (mostly theoretical).
  100     BOOL done, done_and_have_an_output_var, make_result_persistent, left_branch_is_true
  101         , left_was_negative, is_pre_op; // BOOL vs. bool benchmarks slightly faster, and is slightly smaller in code size (or maybe it's cp1's int vs. char that shrunk it).
  102     ExprTokenType *circuit_token, *this_postfix, *p_postfix;
  103     Var *sym_assign_var, *temp_var;
  104 
  105     // v1.0.44.06: EXPR_SMALL_MEM_LIMIT is the means by which _alloca() is used to boost performance a
  106     // little by avoiding the overhead of malloc+free for small strings.  The limit should be something
  107     // small enough that the chance that even 10 of them would cause stack overflow is vanishingly small
  108     // (the program is currently compiled to allow stack to expand anyway).  Even in a worst-case
  109     // scenario where an expression is composed entirely of functions and they all need to use this
  110     // limit of stack space, there's a practical limit on how many functions you can call in an
  111     // expression due to MAX_TOKENS (probably around MAX_TOKENS / 3).
  112     #define EXPR_SMALL_MEM_LIMIT 4097 // The maximum size allowed for an item to qualify for alloca.
  113     #define EXPR_ALLOCA_LIMIT 40000  // The maximum amount of alloca memory for all items.  v1.0.45: An extra precaution against stack stress in extreme/theoretical cases.
  114 
  115     // For each item in the postfix array: if it's an operand, push it onto stack; if it's an operator or
  116     // function call, evaluate it and push its result onto the stack.  SYM_INVALID is the special symbol
  117     // that marks the end of the postfix array.
  118     for (this_postfix = postfix; this_postfix->symbol != SYM_INVALID; ++this_postfix) // Using pointer vs. index (e.g. postfix[i]) reduces OBJ code size by ~122 and seems to perform at least as well.
  119     {
  120         // Set default early to simplify the code.  All struct members are needed: symbol_type (e.g. in
  121         // cases such as this token being an ordinary operand), circuit_token to preserve loadtime info,
  122         // buf for SYM_DYNAMIC, and marker (in cases such as literal strings).  Also, this_token is used
  123         // almost everywhere further below in preference to this_postfix because:
  124         // 1) The various SYM_ASSIGN_* operators (e.g. SYM_ASSIGN_CONCAT) are changed to different operators
  125         //    to simplify the code.  So must use the changed/new value in this_token, not the original value in
  126         //    this_postfix.
  127         // 2) Using a particular variable very frequently might help compiler to optimize that variable to
  128         //    generate faster code.
  129         ExprTokenType &this_token = *(ExprTokenType *)_alloca(sizeof(ExprTokenType)); // Saves a lot of stack space, and seems to perform just as well as something like the following (at the cost of ~82 byte increase in OBJ code size): ExprTokenType &this_token = new_token[new_token_count++]  // array size MAX_TOKENS
  130         this_token = *this_postfix; // Struct copy. See comment section above.
  131 
  132         // At this stage, operands in the postfix array should be SYM_OPERAND, SYM_STRING, or SYM_DYNAMIC.
  133         // But all are checked since that operation is just as fast:
  134         if (IS_OPERAND(this_token.symbol)) // If it's an operand, just push it onto stack for use by an operator in a future iteration.
  135         {
  136             if (this_token.symbol == SYM_DYNAMIC) // CONVERTED HERE/EARLY TO SOMETHING *OTHER* THAN SYM_DYNAMIC so that no later stages need any handling for them as operands. SYM_DYNAMIC is quite similar to SYM_FUNC/BIF in this respect.
  137             {
  138                 if (SYM_DYNAMIC_IS_DOUBLE_DEREF(this_token)) // Double-deref such as Array%i%.
  139                 {
  140                     // Start off by looking for the first deref.
  141                     deref = (DerefType *)this_token.var; // MUST BE DONE PRIOR TO OVERWRITING MARKER/UNION BELOW.
  142                     cp = this_token.buf; // Start at the beginning of this arg's text.
  143                     size_t var_name_length = 0;
  144                     
  145                     this_token.marker = _T("");         // Set default in case of early goto.  Must be done after above.
  146                     this_token.symbol = SYM_STRING; //
  147 
  148                     if (deref->marker == cp && !cp[deref->length] && (deref+1)->is_function // %soleDeref%()
  149                         && deref->var->HasObject()) // It's an object; implies var->Type() == VAR_NORMAL.
  150                     {
  151                         // Push the object, not the var, in case this variable is modified in the
  152                         // parameter list of this function call (which is evaluated prior to SYM_FUNC).
  153                         this_token.symbol = SYM_OBJECT;
  154                         this_token.object = deref->var->Object();
  155                         this_token.object->AddRef();
  156                         goto push_this_token;
  157                     }
  158                     // Otherwise, it could still be %var%() where var contains a string which is too
  159                     // long for the section below to handle.  Supporting that case doesn't seem
  160                     // worthwhile for the following reasons:
  161                     //  1) It would require some logic in the above section to allocate memory for the
  162                     //     string, keeping in mind that deref->var could be VAR_CLIPBOARD/BUILTIN which
  163                     //     can't be pushed directly onto the stack.
  164                     //  2) Pushing a var onto the stack would cause inconsistent behaviour if that var
  165                     //     is modified within the function call's parameter list (although that already
  166                     //     causes inconsistent behaviour if the var appears multiple times in the list).
  167                     //  3) Even if the %var%() case was supported, we don't know at this point whether
  168                     //     xxx%var%() or similar will exceed MAX_VAR_NAME_LENGTH, which means one more
  169                     //     inconsistency.
  170                     //  4) The only benefit of supporting long strings is consistency with regard to
  171                     //     __Call, which will probably only be used for debugging.  Future changes to
  172                     //     error-handling in expressions could supersede that.
  173 
  174                     // Loadtime validation has ensured that none of these derefs are function-calls
  175                     // (i.e. deref->is_function is alway false) with the possible exception of the final
  176                     // deref (the one with a NULL marker), which can be a function-call if this
  177                     // SYM_DYNAMIC is a dynamic call. Other than this, loadtime logic seems incapable
  178                     // of producing function-derefs inside something that would later be interpreted
  179                     // as a double-deref.
  180                     for (; deref->marker; ++deref)  // A deref with a NULL marker terminates the list. "deref" was initialized higher above.
  181                     {
  182                         // FOR EACH DEREF IN AN ARG (if we're here, there's at least one):
  183                         // Copy the chars that occur prior to deref->marker into the buffer:
  184                         for (; cp < deref->marker && var_name_length < MAX_VAR_NAME_LENGTH; left_buf[var_name_length++] = *cp++);
  185                         if (var_name_length >= MAX_VAR_NAME_LENGTH && cp < deref->marker) // The variable name would be too long!
  186                             goto double_deref_fail; // For simplicity and in keeping with the tradition that expressions generally don't display runtime errors, just treat it as a blank.
  187                         // Now copy the contents of the dereferenced var.  For all cases, aBuf has already
  188                         // been verified to be large enough, assuming the value hasn't changed between the
  189                         // time we were called and the time the caller calculated the space needed.
  190                         if (deref->var->Get() > (VarSizeType)(MAX_VAR_NAME_LENGTH - var_name_length)) // The variable name would be too long!
  191                             goto double_deref_fail; // For simplicity and in keeping with the tradition that expressions generally don't display runtime errors, just treat it as a blank.
  192                         var_name_length += deref->var->Get(left_buf + var_name_length);
  193                         // Finally, jump over the dereference text:
  194                         cp += deref->length;
  195                     }
  196 
  197                     // Copy any chars that occur after the final deref into the buffer:
  198                     for (; *cp && var_name_length < MAX_VAR_NAME_LENGTH; left_buf[var_name_length++] = *cp++);
  199                     if (var_name_length >= MAX_VAR_NAME_LENGTH && *cp // The variable name would be too long!
  200                         || !var_name_length && !deref->is_function) // It resolves to an empty string (e.g. a simple dynamic var like %Var% where Var is blank) but isn't a dynamic function call, which would trigger g_MetaObject.__Call().
  201                         goto double_deref_fail; // For simplicity and in keeping with the tradition that expressions generally don't display runtime errors, just treat it as a blank.
  202 
  203                     // Terminate the buffer, even if nothing was written into it:
  204                     left_buf[var_name_length] = '\0';
  205 
  206                     // v1.0.47.06 dynamic function calls: As a result of a prior loop, deref = the null-marker
  207                     // deref which terminates the deref list. is_function is set by the infix processing code.
  208                     if (deref->is_function)
  209                     {   
  210                         // Since the parameter list is evaluated between SYM_DYNAMIC and its corresponding
  211                         // SYM_FUNC, it is possible for a recursive script to re-evaluate this token a second
  212                         // time prior to actually calling the function the first time.  With the old approach
  213                         // of setting the SYM_FUNC's deref->func in this section, such recursion could cause
  214                         // the wrong function to be called.  Instead, push the function name onto the stack
  215                         // to await SYM_FUNC:
  216                         this_token.symbol = SYM_STRING;
  217                         this_token.marker = _tcscpy(talloca(var_name_length + 1), left_buf);
  218                         goto push_this_token;
  219                     }
  220 
  221                     // In v1.0.31, FindOrAddVar() vs. FindVar() is called below to support the passing of non-existent
  222                     // array elements ByRef, e.g. Var:=MyFunc(Array%i%) where the MyFunc function's parameter is
  223                     // defined as ByRef, would effectively create the new element Array%i% if it doesn't already exist.
  224                     // Since at this stage we don't know whether this particular double deref is to be sent as a param
  225                     // to a function, or whether it will be byref, this is done unconditionally for all double derefs
  226                     // since it seems relatively harmless to create a blank variable in something like var := Array%i%
  227                     // (though it will produce a runtime error if the double resolves to an illegal variable name such
  228                     // as one containing spaces).
  229                     if (   !(temp_var = g_script.FindOrAddVar(left_buf, var_name_length))   )
  230                     {
  231                         // Above already displayed the error.  As of v1.0.31, this type of error is displayed and
  232                         // causes the current thread to terminate, which seems more useful than the old behavior
  233                         // that tolerated anything in expressions.
  234                         goto abort;
  235                     }
  236                     // Otherwise, var was found or created.
  237                     this_token.var = temp_var;
  238                 } // Double-deref.
  239                 //else: It's a built-in variable or potential environment variable.
  240 
  241                 // Check if it's a normal variable rather than a built-in or environment variable.
  242                 // This happens when g_NoEnv==FALSE.
  243                 switch (this_token.var->Type())
  244                 {
  245                 case VAR_NORMAL:
  246                     if (g_NoEnv || this_token.var->HasContents()) // v1.0.46.07: It's not an environment variable.
  247                     {
  248                         this_token.symbol = SYM_VAR; // The fact that a SYM_VAR operand is always VAR_NORMAL (with one limited exception) is relied upon in several places such as built-in functions.
  249                         goto push_this_token;
  250                     }
  251                     break;
  252                 case VAR_BUILTIN: // v1.0.48.02: Ensure it's VAR_BUILTIN prior to below because mBIV is a union with mCapacity.
  253                     if (this_token.var->mBIV == BIV_LoopIndex) // v1.0.48.01: Improve performance of A_Index by treating it as an integer rather than a string in expressions (avoids conversions to/from strings).
  254                     {
  255                         this_token.value_int64 = g->mLoopIteration;
  256                         this_token.symbol = SYM_INTEGER;
  257                         goto push_this_token;
  258                     }
  259                     if (this_token.var->mBIV == BIV_EventInfo) // v1.0.48.02: A_EventInfo is used often enough in performance-sensitive numeric contexts to seem worth special treatment like A_Index; e.g. LV_GetText(RowText, A_EventInfo) or RegisterCallback()'s A_EventInfo.
  260                     {
  261                         this_token.value_int64 = g->EventInfo;
  262                         this_token.symbol = SYM_INTEGER;
  263                         goto push_this_token;
  264                     }
  265                     // ABOVE: Goto's and simple assignments (like the SYM_INTEGER ones above) are only a few
  266                     // bytes in code size, so it would probably cost more than it's worth in performance
  267                     // and code size to merge them into a code section shared by all of the above.  Although
  268                     // each comparison "this_token.var->mBIV == BIV_xxx" is surprisingly large in OBJ size,
  269                     // the resulting EXE does not reflect this: even 27 such comparisons and sections (all
  270                     // to a different BIV) don't increase the uncompressed EXE size.
  271                     //
  272                     // OTHER CANDIDATES FOR THE ABOVE:
  273                     // A_TickCount: Usually not performance-critical.
  274                     // A_GuiWidth/Height: Maybe not used in expressions often enough.
  275                     // A_GuiX/Y: Not performance-critical and/or too rare: Popup menu, DropFiles, PostMessage's coords.
  276                     // A_Gui: Hard to say.
  277                     // A_LastError: Seems too rare to justify the code size and loss of performance here.
  278                     // A_Msec: Would help but it's probably rarely used; probably has poor granularity, not likely to be better than A_TickCount.
  279                     // A_TimeIdle/Physical: These are seldom performance-critical.
  280                     //
  281                     // True, False, A_IsUnicode and A_PtrSize are handled in ExpressionToPostfix().
  282                     // Since their values never change at run-time, they are replaced at load-time
  283                     // with the appropriate SYM_INTEGER value.
  284                     //
  285                     break; // case VAR_BUILTIN
  286                 }
  287                 // Otherwise, it's an environment variable, built-in variable, or normal variable of zero-length
  288                 result_size = this_token.var->Get() + 1; // Get() is used even for environment vars because it has a cache that improves their performance.
  289                 if (result_size == 1)
  290                 {
  291                     if (this_token.var->Type() == VAR_NORMAL) // It's an empty variable, so treated as a non-environment (normal) var.
  292                     {
  293                         // The following is done here rather than during infix creation/tokenizing because
  294                         // 1) It's more correct because it's conceivable that some part of the expression
  295                         //    that has already been evaluated before this_token has newly made an environment
  296                         //    variable blank or non-blank, which should be detected here (i.e. only at the
  297                         //    last possible moment).  For example, a function might have the side-effect of
  298                         //    altering an environment variable.
  299                         // 2) It performs better because Get()'s environment variable cache is most effective
  300                         //    when each size-Get() is followed immediately by a contents-Get() for the same
  301                         //    variable.
  302                         // Must make empty variables that aren't environment variables into SYM_VAR so that
  303                         // they can be passed by reference into functions, their address can be taken with
  304                         // the '&' operator, and so that they can be the lvalue for an assignment.
  305                         // Environment variables aren't supported for any of that because it would be silly
  306                         // in most cases, and would probably complicate the code far more than its worth.
  307                         this_token.symbol = SYM_VAR; // The fact that a SYM_VAR operand is always VAR_NORMAL (with one limited exception) is relied upon in several places such as built-in functions.
  308                     }
  309                     else // It's a built-in variable that's blank.
  310                     {
  311                         this_token.marker = _T("");
  312                         this_token.symbol = SYM_STRING;
  313                     }
  314                     goto push_this_token;
  315                 }
  316                 // Otherwise, it's neither an empty string nor a normal variable.
  317                 // It must be an environment variable or built-in variable. Need some memory to store it.
  318                 // The following section is similar to that in the make_result_persistent section further
  319                 // below.  So maintain them together and see it for more comments.
  320                 // Must cast to int to avoid loss of negative values:
  321                 if (result_size <= (int)(aDerefBufSize - (target - aDerefBuf))) // There is room at the end of our deref buf, so use it.
  322                 {
  323                     // Point result to its new, more persistent location:
  324                     result = target;
  325                     target += result_size; // Point it to the location where the next string would be written.
  326                 }
  327                 else if (result_size < EXPR_SMALL_MEM_LIMIT && alloca_usage < EXPR_ALLOCA_LIMIT) // See comments at EXPR_SMALL_MEM_LIMIT.
  328                 {
  329                     result = (LPTSTR)talloca(result_size);
  330                     alloca_usage += result_size; // This might put alloca_usage over the limit by as much as EXPR_SMALL_MEM_LIMIT, but that is fine because it's more of a guideline than a limit.
  331                 }
  332                 else // Need to create some new persistent memory for our temporary use.
  333                 {
  334                     if (mem_count == MAX_EXPR_MEM_ITEMS // No more slots left (should be nearly impossible).
  335                         || !(mem[mem_count] = tmalloc(result_size)))
  336                     {
  337                         LineError(ERR_OUTOFMEM, FAIL, this_token.var->mName);
  338                         goto abort;
  339                     }
  340                     // Point result to its new, more persistent location:
  341                     result = mem[mem_count];
  342                     ++mem_count; // Must be done last.
  343                 }
  344                 this_token.var->Get(result); // MUST USE "result" TO AVOID OVERWRITING MARKER/VAR UNION.
  345                 this_token.marker = result;  // Must be done after above because marker and var overlap in union.
  346                 this_token.buf = NULL; // Indicate that this SYM_OPERAND token LACKS a pre-converted binary integer.
  347                 this_token.symbol = SYM_OPERAND; // Generic operand so that it can later be interpreted as a number (if it's numeric).
  348             } // if (this_token.symbol == SYM_DYNAMIC)
  349             goto push_this_token;
  350         } // if (IS_OPERAND(this_token.symbol))
  351 
  352         if (this_token.symbol == SYM_FUNC) // A call to a function (either built-in or defined by the script).
  353         {
  354             Func *func = this_token.deref->func;
  355             actual_param_count = this_token.deref->param_count; // For performance.
  356             if (actual_param_count > stack_count) // Prevent stack underflow (probably impossible if actual_param_count is accurate).
  357                 goto abnormal_end;
  358             // Adjust the stack early to simplify.  Above already confirmed that the following won't underflow.
  359             // Pop the actual number of params involved in this function-call off the stack.
  360             stack_count -= actual_param_count; // Now stack[stack_count] is the leftmost item in an array of function-parameters, which simplifies processing later on.
  361             ExprTokenType **params = stack + stack_count;
  362 
  363             if (!func)
  364             {
  365                 // This is a dynamic function call.
  366                 if (!stack_count) // SYM_DYNAMIC should have pushed a function name or reference onto the stack, but a syntax error may still cause this condition.
  367                     goto abnormal_end;
  368                 stack_count--;
  369                 func = TokenToFunc(*stack[stack_count]); // Supports function names and function references.
  370                 if (!func)
  371                 {
  372                     // This isn't a function name or reference, but it could be an object emulating
  373                     // a function reference.  Additionally, we want something like %emptyvar%() to
  374                     // invoke g_MetaObject, so this part is done even if stack[stack_count] is not
  375                     // an object.  To "call" the object/value, we need to insert an empty method
  376                     // name between the object/value and the parameter list.  There should always
  377                     // be room for this since the maximum number of operands at any one time <=
  378                     // postfix token count < infix token count < MAX_TOKENS == _countof(stack).
  379                     // That is, each extra (SYM_OPAREN, SYM_COMMA or SYM_CPAREN) token in infix
  380                     // effectively reserves one stack slot.
  381                     if (actual_param_count)
  382                     {
  383                         memmove(params + 1, params, actual_param_count * sizeof(ExprTokenType *));
  384                         high_water_mark++; // If this isn't done and the last param is an object, it won't be released.
  385                     }
  386                     // Insert an empty string:
  387                     params[0] = (ExprTokenType *)_alloca(sizeof(ExprTokenType));
  388                     params[0]->symbol = SYM_STRING;
  389                     params[0]->marker = _T("");
  390                     params--; // Include the object, which is already in the right place.
  391                     actual_param_count += 2;
  392                     extern ExprOpFunc g_ObjCall;
  393                     func = &g_ObjCall;
  394                 }
  395                 // Above has set func to a non-NULL value, but still need to verify there are enough params.
  396                 // Although passing too many parameters is useful (due to the limitations of variadic calls),
  397                 // passing too few parameters (and treating the missing ones as optional) seems a little
  398                 // inappropriate because it would allow the function's caller to second-guess the function's
  399                 // designer (the designer could provide a default value if a parameter is capable of being
  400                 // omitted). Another issue might be misbehavior by built-in functions that assume that the
  401                 // minimum number of parameters are present due to prior validation.  So either all the
  402                 // built-in functions would have to be reviewed, or the minimum would have to be enforced
  403                 // for them but not user-defined functions, which is inconsistent.  Finally, allowing too-
  404                 // few parameters seems like it would reduce the ability to detect script bugs at runtime.
  405                 // Traditionally, expressions don't display any runtime errors.  So if the function is being
  406                 // called incorrectly by the script, the expression is aborted like it would be for other
  407                 // syntax errors:
  408                 if (actual_param_count < func->mMinParams && this_token.deref->is_function != DEREF_VARIADIC)
  409                     goto abnormal_end;
  410             }
  411             
  412             // The following two steps are now done inside Func::Call:
  413             //this_token.symbol = SYM_INTEGER; // Set default return type so that functions don't have to do it if they return INTs.
  414             //this_token.marker = func.mName;  // Inform function of which built-in function called it (allows code sharing/reduction). Can't use circuit_token because it's value is still needed later below.
  415             this_token.buf = left_buf;       // mBIF() can use this to store a string result, and for other purposes.
  416             
  417             // BACK UP THE CIRCUIT TOKEN (it's saved because it can be non-NULL at this point; verified
  418             // through code review).  Currently refers to the same memory as mem_to_free via union.
  419             circuit_token = this_token.circuit_token;
  420             this_token.mem_to_free = NULL; // Init to detect whether the called function allocates it (i.e. we're overloading it with a new purpose).  It's no longer necessary to back up & restore the previous value in circuit_token because circuit_token is used only when a result is about to get pushed onto the stack.
  421             // RESIST TEMPTATIONS TO OPTIMIZE CIRCUIT_TOKEN by passing output_var as circuit_token/mem_to_free
  422             // when done==true (i.e. the built-in function could then assign directly to output_var).
  423             // It doesn't help performance at all except for a mere 10% or less in certain fairly rare cases.
  424             // More importantly, it hurts maintainability because it makes RegExReplace() more complicated
  425             // than it already is, and worse: each BIF would have to check that output_var doesn't overlap
  426             // with its input/source strings because if it does, the function must not initialize a default
  427             // in output_var before starting (and avoiding this would further complicate the code).
  428             // Here is the crux of the abandoned approach: Any function that wishes to pass memory back to
  429             // us via mem_to_free: When mem_to_free!=NULL, that function MUST INSTEAD: 1) Turn that
  430             // memory over to output_var via AcceptNewMem(); 2) Set mem_to_free to NULL to indicate to
  431             // us that it is a user of mem_to_free.
  432             
  433             UDFCallInfo func_call;
  434             // Call the user-defined or built-in function.  Func::Call takes care of variadic parameter
  435             // lists and stores local var backups for UDFs in func_call.  Once func_call goes out of scope,
  436             // its destructor calls Var::FreeAndRestoreFunctionVars() if appropriate.
  437             if (!func->Call(func_call, aResult, this_token, params, actual_param_count
  438                                     , this_token.deref->is_function == DEREF_VARIADIC))
  439             {
  440                 // Take a shortcut because for backward compatibility, ACT_ASSIGNEXPR (and anything else
  441                 // for that matter) is being aborted by this type of early return (i.e. if there's an
  442                 // output_var, its contents are left as-is).  In other words, this expression will have
  443                 // no result storable by the outside world.
  444                 if (aResult != OK) // i.e. EARLY_EXIT or FAIL
  445                     result_to_return = NULL; // Use NULL to inform our caller that this thread is finished (whether through normal means such as Exit or a critical error).
  446                     // Above: The callers of this function know that the value of aResult (which already contains the
  447                     // reason for early exit) should be considered valid/meaningful only if result_to_return is NULL.
  448                 goto normal_end_skip_output_var; // output_var is left unchanged in these cases.
  449             }
  450 
  451 #ifdef CONFIG_DEBUGGER
  452             // See PostExecFunctionCall() itself for comments.
  453             g_Debugger.PostExecFunctionCall(this);
  454 #endif
  455             g_script.mCurrLine = this; // For error-reporting.
  456 
  457             if (IS_NUMERIC(this_token.symbol) || this_token.symbol == SYM_OBJECT) // No need for make_result_persistent or early Assign(). Any numeric result can be considered final because it's already stored in permanent memory (namely the token itself).  L31: This also applies to SYM_OBJECT.
  458             {
  459                 // For code simplicity, the optimization for numeric results is done at a later stage.
  460                 // Additionally, this_token.mem_to_free is assumed to be NULL since the result is not
  461                 // a string; i.e. the function would've freed any memory it allocated without our help.
  462                 this_token.circuit_token = circuit_token; // Restore it to its original value.
  463                 goto push_this_token;
  464             }
  465             //else it's a string, which might need to be moved to persistent memory further below.
  466                 ASSERT(this_token.symbol == SYM_STRING);
  467             
  468             #define EXPR_IS_DONE (!stack_count && this_postfix[1].symbol == SYM_INVALID) // True if we've used up the last of the operators & operands.
  469             done = EXPR_IS_DONE;
  470 
  471             // v1.0.45: If possible, take a shortcut for performance.  Doing it this way saves at least
  472             // two memcpy's (one into deref buffer and then another back into the output_var by
  473             // ACT_ASSIGNEXPR itself).  In some cases is also saves from having to expand the deref
  474             // buffer as well as the output_var (since it's current memory might be too small to hold
  475             // the new memory block). Thus we give it a new block directly to avoid all of that.
  476             // This should be a big boost to performance when long strings are involved.
  477             Var *internal_output_var;
  478             if (done) // i.e. we've now produced the final result.
  479             {
  480                 if (mActionType == ACT_EXPRESSION) // Isolated expression: Outermost function call's result will be ignored, so no need to store it.
  481                 {
  482                     if (this_token.mem_to_free)
  483                         free(this_token.mem_to_free); // Don't bother putting it into mem[].
  484                     goto normal_end_skip_output_var; // No output_var is possible for ACT_EXPRESSION.
  485                 }
  486                 internal_output_var = output_var; // NULL unless this is ACT_ASSIGNEXPR.
  487             }
  488             // It's fairly rare that the following optimization is even applicable because it requires
  489             // an assignment *internal* to an expression, such as "if not var:=func()", or "a:=b, c:=func()".
  490             // But it seems best to optimize these cases so that commas aren't penalized.
  491             else if (this_postfix[1].symbol == SYM_ASSIGN  // Next operation is ":=".
  492                     && stack_count && stack[stack_count-1]->symbol == SYM_VAR // i.e. let the next iteration handle errors instead of doing it here.  Further below relies on this having been checked.
  493                     && stack[stack_count-1]->var->Type() == VAR_NORMAL) // Don't do clipboard here because: 1) AcceptNewMem() doesn't support it; 2) Could probably use Assign() and then make its result be a newly added mem_count item, but the code complexity doesn't seem worth it given the rarity.
  494                 internal_output_var = stack[stack_count-1]->var;
  495             else
  496                 internal_output_var = NULL;
  497             
  498             // RELIES ON THE IS_NUMERIC() CHECK above having been done first.
  499             result = this_token.marker; // Marker can be used because symbol will never be SYM_VAR in this case.
  500 
  501             if (internal_output_var
  502                 && !(g->CurrentFunc == func && internal_output_var->IsNonStaticLocal())) // Ordered for short-circuit performance.
  503                 // Above line is a fix for v1.0.45.03: It detects whether output_var is among the variables
  504                 // that are about to be restored from backup.  If it is, we can't assign to it now
  505                 // because it's currently a local that belongs to the instance we're in the middle of
  506                 // calling; i.e. it doesn't belong to our instance (which is beneath it on the call stack
  507                 // until after the restore-from-backup is done later below).  And we can't assign "result"
  508                 // to it *after* the restore because by then result may have been freed (if it happens to be
  509                 // a local variable too).  Therefore, continue on to the normal method, which will check
  510                 // whether "result" needs to be stored in more persistent memory.
  511             {
  512                 // Check if the called function allocated some memory for its result and turned it over to us.
  513                 // In most cases, the string stored in mem_to_free (if it has been set) is the same address as
  514                 // this_token.marker (i.e. what is named "result" further below), because that's what the
  515                 // built-in functions are normally using the memory for.
  516                 if (this_token.mem_to_free == this_token.marker) // marker is checked in case caller alloc'd mem but didn't use it as its actual result.  Relies on the fact that marker is never NULL.
  517                 {
  518                     // So now, turn over responsibility for this memory to the variable. The called function
  519                     // is responsible for having stored the length of what's in the memory as an overload of
  520                     // this_token.buf, but only when that memory is the result (currently might always be true).
  521                     // AcceptNewMem() will shrink the memory for us, via _expand(), if there's a lot of
  522                     // extra/unused space in it.
  523                     internal_output_var->AcceptNewMem(this_token.mem_to_free, this_token.marker_length);
  524                 }
  525                 else
  526                 {
  527                     result_length = (VarSizeType)_tcslen(result);
  528                     // 1.0.46.06: If the UDF has stored its result in its deref buffer, take possession
  529                     // of that buffer, which saves a memcpy of a potentially huge string.  The cost
  530                     // of this is that if there are any other UDF-calls pending after this one, the
  531                     // code in their bodies will have to create another deref buffer if they need one.
  532                     if (result == sDerefBuf && result_length >= MAX_ALLOC_SIMPLE) // Result is in their buffer and it's longer than what can fit in a SimpleHeap variable (avoids wasting SimpleHeap memory).
  533                     {
  534                         internal_output_var->AcceptNewMem(result, result_length);
  535                         NULLIFY_S_DEREF_BUF // Force any UDFs called subsequently by us to create a new deref buffer because this one was just taken over by a variable.
  536                     }
  537                     else
  538                     {
  539                         // v1.0.45: This mode improves performance by avoiding the need to copy the result into
  540                         // more persistent memory, then avoiding the need to copy it into the defer buffer (which
  541                         // also avoids the possibility of needing to expand that buffer).
  542                         if (!internal_output_var->Assign(this_token.marker)) // Marker can be used because symbol will never be SYM_VAR in this case.
  543                             goto abort;                                      // ALSO: Assign() contains an optimization that avoids actually doing the mem-copying if output_var is being assigned to itself (which can happen in cases like RegExMatch()).
  544                     }
  545                 }
  546                 if (done)
  547                     goto normal_end_skip_output_var; // No need to restore circuit_token because the expression is finished.
  548                 // Next operation is ":=" and above has verified the target is SYM_VAR and VAR_NORMAL.
  549                 --stack_count; // STACK_POP;
  550                 this_token.circuit_token = (++this_postfix)->circuit_token; // Must be done AFTER above. Old, somewhat obsolete comment: this_postfix.circuit_token should have been NULL prior to this because the final right-side result of an assignment shouldn't be the last item of an AND/OR/IFF's left branch. The assignment itself would be that.
  551                 this_token.var = internal_output_var; // Make the result a variable rather than a normal operand so that its
  552                 this_token.symbol = SYM_VAR; // address can be taken, and it can be passed ByRef. e.g. &(x:=1)
  553                 goto push_this_token;
  554             }
  555             // Otherwise, there's no output_var or the expression isn't finished yet, so do normal processing.
  556                 
  557             make_result_persistent = true; // Set default.
  558 
  559             // RESTORE THE CIRCUIT TOKEN (after handling what came back inside it):
  560             if (this_token.mem_to_free) // The called function allocated some memory and turned it over to us.
  561             {
  562                 if (mem_count == MAX_EXPR_MEM_ITEMS) // No more slots left (should be nearly impossible).
  563                 {
  564                     LineError(ERR_OUTOFMEM, FAIL, func->mName);
  565                     goto abort;
  566                 }
  567                 if (this_token.mem_to_free == this_token.marker) // mem_to_free is checked in case caller alloc'd mem but didn't use it as its actual result.
  568                 {
  569                     make_result_persistent = false; // Override the default set higher above.
  570                 }
  571                 // Mark it to be freed at the time we return.
  572                 mem[mem_count++] = this_token.mem_to_free;
  573             }
  574             //else this_token.mem_to_free==NULL, so the BIF just called didn't allocate memory to give to us.
  575             this_token.circuit_token = circuit_token; // Restore it to its original value.
  576 
  577             // Empty strings are returned pretty often by UDFs, such as when they don't use "return"
  578             // at all.  Therefore, handle them fully now, which should improve performance (since it
  579             // avoids all the other checking later on).  It also doesn't hurt code size because this
  580             // check avoids having to check for empty string in other sections later on.
  581             if (!*this_token.marker) // Various make-persistent sections further below may rely on this check.
  582             {
  583                 this_token.marker = _T(""); // Ensure it's a constant memory area, not a buf that might get overwritten soon.
  584                 this_token.symbol = SYM_OPERAND; // SYM_OPERAND vs. SYM_STRING probably doesn't matter in the case of empty string, but it's used for consistency with what the other UDF handling further below does.
  585                 this_token.buf = NULL; // Indicate that this SYM_OPERAND token LACKS a pre-converted binary integer.
  586                 goto push_this_token; // For code simplicity, the optimization for numeric results is done at a later stage.
  587             }
  588 
  589             if (func->mIsBuiltIn)
  590             {
  591                 // Since above didn't goto, "result" is not SYM_INTEGER/FLOAT/VAR, and not "".  Therefore, it's
  592                 // either a pointer to static memory (such as a constant string), or more likely the small buf
  593                 // we gave to the BIF for storing small strings.  For simplicity assume it's the buf, which is
  594                 // volatile and must be made persistent if called for below.
  595                 if (make_result_persistent) // At this stage, this means that the above wasn't able to determine its correct value yet.
  596                     make_result_persistent = !done;
  597             }
  598             else // It's not a built-in function.
  599             {
  600                 // Since above didn't goto:
  601                 // The result just returned may need to be copied to a more persistent location.  This is done right
  602                 // away if the result is the contents of a local variable (since all locals are about to be freed
  603                 // and overwritten), which is assumed to be the case if it's not in the new deref buf because it's
  604                 // difficult to distinguish between when the function returned one of its own local variables
  605                 // rather than a global or a string/numeric literal).  The only exceptions are covered below.
  606                 // Old method, not necessary to be so thorough because "return" always puts its result as the
  607                 // very first item in its deref buf.  So this is commented out in favor of the line below it:
  608                 //if (result < sDerefBuf || result >= sDerefBuf + sDerefBufSize)
  609                 if (result != sDerefBuf) // Not in their deref buffer (yields correct result even if sDerefBuf is NULL; also, see above.)
  610                     // In this case, the result must be assumed to be one of their local variables (since there's
  611                     // no way to distinguish between that and a literal string such as "abc"?). So it should be
  612                     // immediately copied since if it's a local, it's about to be freed.
  613                     make_result_persistent = true;
  614                 else // The result must be in their deref buffer, perhaps due to something like "return x+3" or "return bif()" on their part.
  615                 {
  616                     make_result_persistent = false; // Set default to be possibly overridden below.
  617                     if (!done) // There are more operators/operands to be evaluated, but if there are no more function calls, we don't have to make it persistent since their deref buf won't be overwritten by anything during the time we need it.
  618                     {
  619                         // Since there's more in the stack or postfix array to be evaluated, and since the return
  620                         // value is in the new deref buffer, must copy result to somewhere non-volatile whenever
  621                         // there's another function-call pending by us.  Note that an empty-string result was
  622                         // already checked and fully handled higher above.
  623                         // If we don't have any more user-defined function calls pending, we can skip the
  624                         // make-persistent section since this deref buffer will not be overwritten during the
  625                         // period we need it.
  626                         for (p_postfix = this_postfix + 1; p_postfix->symbol != SYM_INVALID; ++p_postfix)
  627                             if (p_postfix->symbol == SYM_FUNC)
  628                             {
  629                                 make_result_persistent = true;
  630                                 break;
  631                             }
  632                     }
  633                     //else done==true, so don't have to make it persistent here because the final stage will
  634                     // copy it from their deref buf into ours (since theirs is only deleted later, by our caller).
  635                     // In this case, leave make_result_persistent set to false.
  636                 } // This is the end of the section that determines the value of "make_result_persistent" for UDFs.
  637             }
  638 
  639             this_token.buf = NULL; // Indicate that this SYM_OPERAND token LACKS a pre-converted binary integer.
  640             this_token.symbol = SYM_OPERAND; // Use generic, not string, so that any operator or function call that uses this result is free to reinterpret it as an integer or float.
  641             if (make_result_persistent) // Both UDFs and built-in functions have ensured make_result_persistent is set.
  642             {
  643                 // BELOW RELIES ON THE ABOVE ALWAYS HAVING VERIFIED AND FULLY HANDLED RESULT BEING AN EMPTY STRING.
  644                 // So now we know result isn't an empty string, which in turn ensures that size > 1 and length > 0,
  645                 // which might be relied upon by things further below.
  646                 result_size = _tcslen(result) + 1; // No easy way to avoid strlen currently. Maybe some future revisions to architecture will provide a length.
  647                 // Must cast to int to avoid loss of negative values:
  648                 if (result_size <= aDerefBufSize - (target - aDerefBuf)) // There is room at the end of our deref buf, so use it.
  649                 {
  650                     // Make the token's result the new, more persistent location:
  651                     this_token.marker = (LPTSTR)tmemcpy(target, result, result_size); // Benches slightly faster than strcpy().
  652                     target += result_size; // Point it to the location where the next string would be written.
  653                 }
  654                 else if (result_size < EXPR_SMALL_MEM_LIMIT && alloca_usage < EXPR_ALLOCA_LIMIT) // See comments at EXPR_SMALL_MEM_LIMIT.
  655                 {
  656                     this_token.marker = (LPTSTR)tmemcpy((LPTSTR)talloca(result_size), result, result_size); // Benches slightly faster than strcpy().
  657                     alloca_usage += result_size; // This might put alloca_usage over the limit by as much as EXPR_SMALL_MEM_LIMIT, but that is fine because it's more of a guideline than a limit.
  658                 }
  659                 else // Need to create some new persistent memory for our temporary use.
  660                 {
  661                     // In real-world scripts the need for additional memory allocation should be quite
  662                     // rare because it requires a combination of worst-case situations:
  663                     // - Called-function's return value is in their new deref buf (rare because return
  664                     //   values are more often literal numbers, true/false, or variables).
  665                     // - We still have more functions to call here (which is somewhat atypical).
  666                     // - There's insufficient room at the end of the deref buf to store the return value
  667                     //   (unusual because the deref buf expands in block-increments, and also because
  668                     //   return values are usually small, such as numbers).
  669                     if (mem_count == MAX_EXPR_MEM_ITEMS // No more slots left (should be nearly impossible).
  670                         || !(mem[mem_count] = tmalloc(result_size)))
  671                     {
  672                         LineError(ERR_OUTOFMEM, FAIL, func->mName);
  673                         goto abort;
  674                     }
  675                     // Make the token's result the new, more persistent location:
  676                     this_token.marker = (LPTSTR)tmemcpy(mem[mem_count], result, result_size); // Benches slightly faster than strcpy().
  677                     ++mem_count; // Must be done last.
  678                 }
  679             }
  680             else // make_result_persistent==false
  681                 this_token.marker = result;
  682 
  683             goto push_this_token;
  684         } // if (this_token.symbol == SYM_FUNC)
  685 
  686         if (this_token.symbol == SYM_IFF_ELSE) // This is encountered when a ternary's condition was found to be false by a prior iteration.
  687         {
  688             if (this_token.circuit_token // This ternary's result is some other ternary's condition (somewhat rare).
  689                 && stack_count) // Prevent underflow (this check might not be necessary; so it's just in case there's a way it can happen).
  690             {
  691                 // To support *cascading* short-circuit when ternary/IFF's are nested inside each other, pop the
  692                 // topmost operand off the stack to modify its circuit_token.  The routine below will then
  693                 // use this as the parent IFF's *condition*, which is an non-operand of sorts because it's
  694                 // used only to determine which branch of an IFF will become the operand/result of this IFF.
  695                 this_token = *STACK_POP; // Struct copy.  Doing it this way is more maintainable than other methods, and is unlikely to perform much worse.
  696                 this_token.circuit_token = this_postfix->circuit_token; // Override the circuit_token that was just set in the line above.
  697                 goto non_null_circuit_token; // Must do this so that it properly evaluates this_postfix as the next ternary's condition.
  698             }
  699             // Otherwise, ignore it because its final result has already been evaluated and pushed onto the
  700             // stack via prior iterations.  In other words, this ELSE branch was the IFF's final result, which
  701             // is now topmost on the stack for use as an operand by a future operator.
  702             continue;
  703         }
  704 
  705         // Since the above didn't goto or continue, this token must be a unary or binary operator.
  706         // Get the first operand for this operator (for non-unary operators, this is the right-side operand):
  707         if (!stack_count) // Prevent stack underflow.  An expression such as -*3 causes this.
  708             goto abnormal_end;
  709         ExprTokenType &right = *STACK_POP;
  710         // Below uses IS_OPERAND rather than checking for only SYM_OPERAND because the stack can contain
  711         // both generic and specific operands.  Specific operands were evaluated by a previous iteration
  712         // of this section.  Generic ones were pushed as-is onto the stack by a previous iteration.
  713         if (!IS_OPERAND(right.symbol)) // Haven't found a way to produce this situation yet, but safe to assume it's possible.
  714             goto abnormal_end;
  715 
  716         // The following check is done after popping "right" off the stack because a prior iteration has set up
  717         // SYM_IFF_THEN to be a unary operator of sorts.
  718         if (this_token.symbol == SYM_IFF_THEN) // This is encountered when a ternary's condition was found to be true by a prior iteration.
  719         {
  720             if (!this_token.circuit_token) // This check is needed for syntax errors such as "1 ? 2" (no matching else) and perhaps other unusual circumstances.
  721                 goto abnormal_end; // Seems best to consider it a syntax error rather than supporting partial functionality (hard to imagine much legitimate need to omit an ELSE).
  722             // SYM_IFF_THEN is encountered only when a previous iteration has determined that the ternary's condition
  723             // is true.  At this stage, the ternary's "THEN" branch has already been evaluated and stored in
  724             // "right".  So skip over its "else" branch (short-circuit) because that doesn't need to be evaluated.
  725             this_postfix = this_token.circuit_token; // The address in any circuit_token always points into the arg's postfix array (never any temporary array or token created here) due to the nature/definition of circuit_token.
  726             // And very soon, the outer loop will skip over the SYM_IFF_ELSE just found above.
  727             right.circuit_token = this_token.circuit_token->circuit_token; // Can be NULL (in fact, it usually is).
  728             this_token = right;   // Struct copy to set things up for push_this_token, which in turn is needed
  729             right.symbol = SYM_INTEGER; // L33: Bugfix.  Since only one reference is counted and this reference is no longer needed, "disable" it.  This avoids calling Release too many times; an alternative would be to call AddRef (if this is an object) and let Release be called later.
  730             goto push_this_token; // (rather than a simple STACK_PUSH(right)) because it checks for *cascading* short circuit in cases where this ternary's result is the boolean condition of another ternary.
  731         }
  732 
  733         switch (this_token.symbol)
  734         {
  735         case SYM_ASSIGN:        // These don't need "right_is_number" to be resolved. v1.0.48.01: Also avoid
  736         case SYM_CONCAT:        // resolving right_is_number for CONCAT because TokenIsPureNumeric() will take
  737         case SYM_ASSIGN_CONCAT: // a long time if the string is very long and consists entirely of digits/whitespace.
  738         case SYM_COMMA:
  739         case SYM_ADDRESS:
  740             break;
  741         default:
  742             // If the operand is still generic/undetermined, find out whether it is a string, integer, or float:
  743             right_is_number = TokenIsPureNumeric(right); // If it's SYM_VAR, it can be the clipboard in this case, but it works even then.
  744         }
  745 
  746         // IF THIS IS A UNARY OPERATOR, we now have the single operand needed to perform the operation.
  747         // The cases in the switch() below are all unary operators.  The other operators are handled
  748         // in the switch()'s default section:
  749         sym_assign_var = NULL; // Set default for use at the bottom of the following switch().
  750         switch (this_token.symbol)
  751         {
  752         case SYM_AND: // These are now unary operators because short-circuit has made them so.  If the AND/OR
  753         case SYM_OR:  // had short-circuited, we would never be here, so this is the right branch of a non-short-circuit AND/OR.
  754             this_token.value_int64 = TokenToBOOL(right, right_is_number);
  755             this_token.symbol = SYM_INTEGER; // Result of AND or OR is always a boolean integer (one or zero).
  756             break;
  757 
  758         case SYM_LOWNOT:  // The operator-word "not".
  759         case SYM_HIGHNOT: // The symbol '!'. Both NOTs are equivalent at this stage because precedence was already acted upon by infix-to-postfix.
  760             this_token.value_int64 = !TokenToBOOL(right, right_is_number);
  761             this_token.symbol = SYM_INTEGER; // Result is always one or zero.
  762             break;
  763 
  764         case SYM_NEGATIVE:  // Unary-minus.
  765             if (right_is_number == PURE_INTEGER)
  766                 this_token.value_int64 = -TokenToInt64(right, TRUE);
  767             else if (right_is_number == PURE_FLOAT)
  768                 this_token.value_double = -TokenToDouble(right, FALSE, TRUE); // Pass FALSE for aCheckForHex since PURE_FLOAT is never hex.
  769             else // String.
  770             {
  771                 // Seems best to consider the application of unary minus to a string, even a quoted string
  772                 // literal such as "15", to be a failure.  UPDATE: For v1.0.25.06, invalid operations like
  773                 // this instead treat the operand as an empty string.  This avoids aborting a long, complex
  774                 // expression entirely just because on of its operands is invalid.  However, the net effect
  775                 // in most cases might be the same, since the empty string is a non-numeric result and thus
  776                 // will cause any operator it is involved with to treat its other operand as a string too.
  777                 // And the result of a math operation on two strings is typically an empty string.
  778                 this_token.marker = _T("");
  779                 this_token.symbol = SYM_STRING;
  780                 break;
  781             }
  782             // Since above didn't "break":
  783             this_token.symbol = right_is_number; // Convert generic SYM_OPERAND into a specific type: float or int.
  784             break;
  785 
  786         case SYM_POST_INCREMENT: // These were added in v1.0.46.  It doesn't seem worth translating them into
  787         case SYM_POST_DECREMENT: // += and -= at load-time or during the tokenizing phase higher above because 
  788         case SYM_PRE_INCREMENT:  // it might introduce precedence problems, plus the post-inc/dec's nature is
  789         case SYM_PRE_DECREMENT:  // unique among all the operators in that it pushes an operand before the evaluation.
  790             is_pre_op = (this_token.symbol >= SYM_PRE_INCREMENT); // Store this early because its symbol will soon be overwritten.
  791             if (right.symbol != SYM_VAR || right_is_number == PURE_NOT_NUMERIC) // Invalid operation.
  792             {
  793                 if (right.symbol == SYM_VAR) // Thus due to the above check, it's a non-numeric target such as ++i when "i" is blank or contains text. This line was fixed in v1.0.46.16.
  794                 {
  795                     right.var->MaybeWarnUninitialized(); // This line should always be reached if the var is uninitialized.
  796                     right.var->Assign(); // If target var contains "" or "non-numeric text", make it blank. Clipboard is also supported here.
  797                     if (is_pre_op)
  798                     {
  799                         // v1.0.46.01: For consistency, it seems best to make the result of a pre-op be a
  800                         // variable whenever a variable came in.  This allows its address to be taken, and it
  801                         // to be passed by reference, and other SYM_VAR behaviors, even if the operation itself
  802                         // produces a blank value.
  803                         // KNOWN LIMITATION: Although this behavior is convenient to have, I realize now
  804                         // that it produces at least one weird effect: whenever a binary operator's operands
  805                         // both use a pre-op on the same variable, or whenever two or more of a function-call's
  806                         // parameters both do a pre-op on the same variable, that variable will have the same
  807                         // value at the time the binary operator or function-call is evaluated.  For example:
  808                         //    y = 1
  809                         //    x = ++y + ++y  ; Yields 6 not 5.
  810                         // However, if you think about the situations anyone would intentionally want to do
  811                         // the above or a function-call with two or more pre-ops in its parameters, it seems
  812                         // so extremely rare that retaining the existing behavior might be superior because of:
  813                         // 1) Convenience: It allows ++x to be passed ByRef, it's address taken.  Less importantly,
  814                         //    it also allows ++++x to work.
  815                         // 2) Backward compatibility: Some existing scripts probably already rely on the fact that
  816                         //    ++x and --x produce an lvalue (though it's undocumented).
  817                         if (right.var->Type() == VAR_NORMAL)
  818                         {
  819                             this_token.var = right.var;  // Make the result a variable rather than a normal operand so that its
  820                             this_token.symbol = SYM_VAR; // address can be taken, and it can be passed ByRef. e.g. &(++x)
  821                             break;
  822                         }
  823                         //else VAR_CLIPBOARD, which is allowed in only when it's the lvalue of an assignment or
  824                         // inc/dec.  So fall through to make the result blank because clipboard isn't allowed as
  825                         // SYM_VAR beyond this point (to simplify the code and improve maintainability).
  826                     }
  827                     //else post_op against non-numeric target-var.  Fall through to below to yield blank result.
  828                 }
  829                 //else target isn't a var.  Fall through to below to yield blank result.
  830                 this_token.marker = _T("");          // Make the result blank to indicate invalid operation
  831                 this_token.symbol = SYM_STRING;  // (assign to non-lvalue or increment/decrement a non-number).
  832                 break;
  833             } // end of "invalid operation" block.
  834 
  835             // DUE TO CODE SIZE AND PERFORMANCE decided not to support things like the following:
  836             // -> ++++i ; This one actually works because pre-ops produce a variable (usable by future pre-ops).
  837             // -> i++++ ; Fails because the first ++ produces an operand that isn't a variable.  It could be
  838             //    supported via a cascade loop here to pull all remaining consecutive post/pre ops out of
  839             //    the postfix array and apply them to "delta", but it just doesn't seem worth it.
  840             // -> --Var++ ; Fails because ++ has higher precedence than --, but it produces an operand that isn't
  841             //    a variable, so the "--" fails.  Things like --Var++ seem pointless anyway because they seem
  842             //    nearly identical to the sub-expression (Var+1)? Anyway, --Var++ could probably be supported
  843             //    using the loop described in the previous example.
  844             delta = (this_token.symbol == SYM_POST_INCREMENT || this_token.symbol == SYM_PRE_INCREMENT) ? 1 : -1;
  845             if (right_is_number == PURE_INTEGER)
  846             {
  847                 this_token.value_int64 = TokenToInt64(right, TRUE);
  848                 right.var->Assign(this_token.value_int64 + delta);
  849             }
  850             else // right_is_number must be PURE_FLOAT because it's the only remaining alternative.
  851             {
  852                 this_token.value_double = TokenToDouble(right, FALSE, TRUE); // Pass FALSE for aCheckForHex since PURE_FLOAT is never hex.
  853                 right.var->Assign(this_token.value_double + delta);
  854             }
  855             if (is_pre_op)
  856             {
  857                 // Push the variable itself so that the operation will have already taken effect for whoever
  858                 // uses this operand/result in the future (i.e. pre-op vs. post-op).
  859                 if (right.var->Type() == VAR_NORMAL)
  860                 {
  861                     this_token.var = right.var;  // Make the result a variable rather than a normal operand so that its
  862                     this_token.symbol = SYM_VAR; // address can be taken, and it can be passed ByRef. e.g. &(++x)
  863                 }
  864                 else // VAR_CLIPBOARD, which is allowed in only when it's the lvalue of an assignment or inc/dec.
  865                 {
  866                     // Clipboard isn't allowed as SYM_VAR beyond this point (to simplify the code and
  867                     // improve maintainability).  So use the new contents of the clipboard as the result,
  868                     // rather than the clipboard itself.
  869                     if (right_is_number == PURE_INTEGER)
  870                         this_token.value_int64 += delta;
  871                     else // right_is_number must be PURE_FLOAT because it's the only alternative remaining.
  872                         this_token.value_double += delta;
  873                     this_token.symbol = right_is_number; // Set the symbol type to match the double or int64 that was already stored higher above.
  874                 }
  875             }
  876             else // Post-inc/dec, so the non-delta version, which was already stored in this_token, should get pushed.
  877                 this_token.symbol = right_is_number; // Set the symbol type to match the double or int64 that was already stored higher above.
  878             break;
  879 
  880         case SYM_ADDRESS: // Take the address of a variable.
  881             if (right.symbol == SYM_VAR) // At this stage, SYM_VAR is always a normal variable, never a built-in one, so taking its address should be safe.
  882             {
  883                 if (right.var->HasObject()) // L31
  884                 {
  885                     this_token.symbol = SYM_INTEGER;
  886                     this_token.value_int64 = (__int64)right.var->Object();
  887                 }
  888                 else
  889                 {
  890                     right.var->DisableCache(); // Once the script take the address of a variable, there's no way to predict when it will make changes to the variable's contents.  So don't allow mContents to get out-of-sync with the variable's binary int/float.
  891                     this_token.symbol = SYM_INTEGER;
  892                     this_token.value_int64 = (__int64)right.var->Contents(); // Contents() vs. mContents to support VAR_CLIPBOARD, and in case mContents needs to be updated by Contents().
  893                 }
  894             }
  895             else if (right.symbol == SYM_OBJECT) // L31
  896             {
  897                 this_token.symbol = SYM_INTEGER;
  898                 this_token.value_int64 = (__int64)right.object;
  899             }
  900             else // Invalid, so make it a localized blank value.
  901             {
  902                 this_token.marker = _T("");
  903                 this_token.symbol = SYM_STRING;
  904             }
  905             break;
  906 
  907         case SYM_DEREF:   // Dereference an address to retrieve a single byte.
  908         case SYM_BITNOT:  // The tilde (~) operator.
  909             if (right_is_number == PURE_NOT_NUMERIC) // String.  Seems best to consider the application of '*' or '~' to a string, even a quoted string literal such as "15", to be a failure.
  910             {
  911                 this_token.marker = _T("");
  912                 this_token.symbol = SYM_STRING;
  913                 break;
  914             }
  915             // Since above didn't "break": right_is_number is PURE_INTEGER or PURE_FLOAT.
  916             right_int64 = TokenToInt64(right, right_is_number==PURE_INTEGER); // Although PURE_FLOAT can't be hex, for simplicity and due to the rarity of encountering a PURE_FLOAT in this case, the slight performance reduction of calling TokenToInt64() is done for both PURE_FLOAT and PURE_INTEGER.
  917             if (this_token.symbol == SYM_BITNOT)
  918             {
  919                 // Note that it is not legal to perform ~, &, |, or ^ on doubles.  Because of this, and also to
  920                 // conform to the behavior of the Transform command, any floating point operand is truncated to
  921                 // an integer above.
  922                 if (right_int64 < 0 || right_int64 > UINT_MAX)
  923                     this_token.value_int64 = ~right_int64;
  924                 else // See comments at TRANS_CMD_BITNOT for why it's done this way:
  925                     this_token.value_int64 = (size_t)(DWORD)~(DWORD)right_int64; // Casting this way avoids compiler warning.
  926             }
  927             else // SYM_DEREF
  928             {
  929                 // Reasons for resolving *Var to a number rather than a single-char string:
  930                 // 1) More consistent with future uses of * that might operate on the address of 2-byte,
  931                 //    4-byte, and 8-byte targets.
  932                 // 2) Performs better in things like ExtractInteger() that would otherwise have to call Asc().
  933                 // 3) Converting it to a one-char string would add no value beyond convenience because script
  934                 //    could do "if (*var = 65)" if it's concerned with avoiding a Chr() call for performance
  935                 //    reasons.  Also, it seems somewhat rare that a script will access a string's characters
  936                 //    one-by-one via the * method because that a parsing loop can already do that more easily.
  937                 // 4) Reduces code size and improves performance (however, the single-char string method would
  938                 //    use _alloca(2) to get some temporary memory, so it wouldn't be too bad in performance).
  939                 //
  940                 // The following does a basic bounds check to prevent crashes due to dereferencing addresses
  941                 // that are obviously bad.  In terms of percentage impact on performance, this seems quite
  942                 // justified.  In the future, could also put a __try/__except block around this (like DllCall
  943                 // uses) to prevent buggy scripts from crashing.  In addition to ruling out the dereferencing of
  944                 // a NULL address, the >255 check also rules out common-bug addresses (I don't think addresses
  945                 // this low can realistically ever be legitimate, but it would be nice to get confirmation).
  946                 // For simplicity and due to rarity, a zero is yielded in such cases rather than an empty string.
  947                 this_token.value_int64 = ((size_t)right_int64 < 4096)
  948                     ? 0 : *(UCHAR *)right_int64; // Dereference to extract one unsigned character, just like Asc().
  949             }
  950             this_token.symbol = SYM_INTEGER; // Must be done only after its old value was used above. v1.0.36.07: Fixed to be SYM_INTEGER vs. right_is_number for SYM_BITNOT.
  951             break;
  952 
  953         default: // NON-UNARY OPERATOR.
  954             // GET THE SECOND (LEFT-SIDE) OPERAND FOR THIS OPERATOR:
  955             if (!stack_count) // Prevent stack underflow.
  956                 goto abnormal_end;
  957             ExprTokenType &left = *STACK_POP; // i.e. the right operand always comes off the stack before the left.
  958             if (!IS_OPERAND(left.symbol)) // Haven't found a way to produce this situation yet, but safe to assume it's possible.
  959                 goto abnormal_end;
  960             
  961             if (this_token.symbol == SYM_COMMA) // This can only be a statement-separator comma, not a function comma, since function commas weren't put into the postfix array.
  962             {
  963                 // Do nothing other than discarding the right-side operand that was just popped off the stack.
  964                 // This collapses the two sub-statements delimited by a given comma into a single result for
  965                 // subsequent uses by another operator.  Unlike C++, the leftmost operand is preserved, not the
  966                 // rightmost.  This is because it's faster to just discard the topmost item on the stack, but
  967                 // more importantly it allows ACT_ASSIGNEXPR, ACT_ADD, and others to work properly.  For example:
  968                 //    Var:=5, Var1:=(Var2:=1, Var3:=2)
  969                 // Without the behavior implemented here, the above would wrongly put Var3's rvalue into Var2.
  970                 // Lexikos: Comments above may be incorrect; if SYM_COMMA is placed immediately after the left
  971                 // branch (i.e. when the topmost item on the stack is the leftmost operand), the rightmost
  972                 // operand is preserved while retaining performance and correct behaviour.  However, the old
  973                 // behaviour is kept in v1 for backward-compatibility.
  974                 // The left operand is popped off the stack (above) and pushed back on so that circuit_token
  975                 // is checked.  Without this, expressions like ((x, y) and z) fail.
  976                 this_token = left; // Struct copy.
  977                 this_token.circuit_token = this_postfix->circuit_token;
  978                 if (this_token.symbol == SYM_OBJECT)
  979                     this_token.object->AddRef();
  980                 goto push_this_token;
  981             }
  982 
  983             if (IS_ASSIGNMENT_EXCEPT_POST_AND_PRE(this_token.symbol)) // v1.0.46: Added support for various assignment operators.
  984             {
  985                 if (left.symbol != SYM_VAR)
  986                 {
  987                     this_token.marker = _T("");          // Make the result blank to indicate invalid operation
  988                     this_token.symbol = SYM_STRING;  // (assign to non-lvalue).
  989                     break; // Equivalent to "goto push_this_token" in this case.
  990                 }
  991                 switch(this_token.symbol)
  992                 {
  993                 case SYM_ASSIGN: // Listed first for performance (it's probably the most common because things like ++ and += aren't expressions when they're by themselves on a line).
  994                     if (!left.var->Assign(right)) // left.var can be VAR_CLIPBOARD in this case.
  995                         goto abort;
  996                     if (left.var->Type() == VAR_CLIPBOARD) // v1.0.46.01: Clipboard is present as SYM_VAR, but only for assign-to-clipboard so that built-in functions and other code sections don't need handling for VAR_CLIPBOARD.
  997                     {
  998                         this_token = right; // Struct copy.  Doing it this way is more maintainable than other methods, and is unlikely to perform much worse.
  999                         this_token.circuit_token = this_postfix->circuit_token; // Override the circuit_token that was just set in the line above.
 1000                     }
 1001                     else
 1002                     {
 1003                         this_token.var = left.var;   // Make the result a variable rather than a normal operand so that its
 1004                         this_token.symbol = SYM_VAR; // address can be taken, and it can be passed ByRef. e.g. &(x:=1)
 1005                     }
 1006                     goto push_this_token;
 1007                 case SYM_ASSIGN_ADD:           this_token.symbol = SYM_ADD; break;
 1008                 case SYM_ASSIGN_SUBTRACT:      this_token.symbol = SYM_SUBTRACT; break;
 1009                 case SYM_ASSIGN_MULTIPLY:      this_token.symbol = SYM_MULTIPLY; break;
 1010                 case SYM_ASSIGN_DIVIDE:        this_token.symbol = SYM_DIVIDE; break;
 1011                 case SYM_ASSIGN_FLOORDIVIDE:   this_token.symbol = SYM_FLOORDIVIDE; break;
 1012                 case SYM_ASSIGN_BITOR:         this_token.symbol = SYM_BITOR; break;
 1013                 case SYM_ASSIGN_BITXOR:        this_token.symbol = SYM_BITXOR; break;
 1014                 case SYM_ASSIGN_BITAND:        this_token.symbol = SYM_BITAND; break;
 1015                 case SYM_ASSIGN_BITSHIFTLEFT:  this_token.symbol = SYM_BITSHIFTLEFT; break;
 1016                 case SYM_ASSIGN_BITSHIFTRIGHT: this_token.symbol = SYM_BITSHIFTRIGHT; break;
 1017                 case SYM_ASSIGN_CONCAT:        this_token.symbol = SYM_CONCAT; break;
 1018                 }
 1019                 // Since above didn't goto or break out of the outer loop, this is an assignment other than
 1020                 // SYM_ASSIGN, so it needs further evaluation later below before the assignment will actually be made.
 1021                 sym_assign_var = left.var; // This tells the bottom of this switch() to do extra steps for this assignment.
 1022             }
 1023 
 1024             // The following section needs done even for assignments such as += because the type of value
 1025             // inside the target variable (integer vs. float vs. string) must be known, to determine how
 1026             // the operation should proceed.
 1027             // Since above didn't goto/break, this is a non-unary operator that needs further processing.
 1028             // If the operand is still generic/undetermined, find out whether it is a string, integer, or float.
 1029             // Fix for v1.0.48.01: For performance, call TokenIsPureNumeric(left) only as a last resort
 1030             // because "left" could be a very long string consisting entirely of digits or whitespace, which
 1031             // would make the call take a long time.  MUST NOT check the value of right_is_number until after
 1032             // checking for SYM_CONCAT because for SYM_CONCAT, right_is_number was left uninitialized at an
 1033             // earlier stage (for performance).
 1034             if (this_token.symbol == SYM_CONCAT || !right_is_number || !(left_is_number = TokenIsPureNumeric(left))) // See comment above.
 1035             {
 1036                 // L31: Handle binary ops supported by objects (= == !=).
 1037                 switch (this_token.symbol)
 1038                 {
 1039                 case SYM_EQUAL:
 1040                 case SYM_EQUALCASE:
 1041                 case SYM_NOTEQUAL:
 1042                     IObject *right_obj = TokenToObject(right);
 1043                     IObject *left_obj = TokenToObject(left);
 1044                     // To support a future "implicit default value" feature, both operands must be objects.
 1045                     // Otherwise, an object operand will be treated as its default value, currently always "".
 1046                     // This is also consistent with unsupported operands such as < and > - i.e. because obj<""
 1047                     // and obj>"" are always false and obj<="" and obj>="" are always true, obj must be "".
 1048                     // When the default value feature is implemented all operators (excluding =, == and !=
 1049                     // if both operands are objects) may use the default value of any object operand.
 1050                     // UPDATE: Above is not done because it seems more intuitive to document the other
 1051                     // comparison operators as unsupported than for (obj == "") to evaluate to true.
 1052                     if (right_obj || left_obj)
 1053                     {
 1054                         this_token.value_int64 = (this_token.symbol != SYM_NOTEQUAL) == (right_obj == left_obj);
 1055                         this_token.symbol = SYM_INTEGER; // Must be set *after* above checks symbol.
 1056                         goto push_this_token;
 1057                     }
 1058                 }
 1059 
 1060                 // Above check has ensured that at least one of them is a string.  But the other
 1061                 // one might be a number such as in 5+10="15", in which 5+10 would be a numerical
 1062                 // result being compared to the raw string literal "15".
 1063                 right_string = TokenToString(right, right_buf);
 1064                 left_string = TokenToString(left, left_buf);
 1065                 result_symbol = SYM_INTEGER; // Set default.  Boolean results are treated as integers.
 1066                 switch(this_token.symbol)
 1067                 {
 1068                 case SYM_EQUAL:     this_token.value_int64 = !((g->StringCaseSense == SCS_INSENSITIVE)
 1069                                         ? _tcsicmp(left_string, right_string)
 1070                                         : lstrcmpi(left_string, right_string)); break; // i.e. use the "more correct mode" except when explicitly told to use the fast mode (v1.0.43.03).
 1071                 case SYM_EQUALCASE: this_token.value_int64 = !_tcscmp(left_string, right_string); break; // Case sensitive.
 1072                 // The rest all obey g->StringCaseSense since they have no case sensitive counterparts:
 1073                 case SYM_NOTEQUAL:  this_token.value_int64 = g_tcscmp(left_string, right_string) ? 1 : 0; break;
 1074                 case SYM_GT:        this_token.value_int64 = g_tcscmp(left_string, right_string) > 0; break;
 1075                 case SYM_LT:        this_token.value_int64 = g_tcscmp(left_string, right_string) < 0; break;
 1076                 case SYM_GTOE:      this_token.value_int64 = g_tcscmp(left_string, right_string) > -1; break;
 1077                 case SYM_LTOE:      this_token.value_int64 = g_tcscmp(left_string, right_string) < 1; break;
 1078 
 1079                 case SYM_CONCAT:
 1080                     // Even if the left or right is "", must copy the result to temporary memory, at least
 1081                     // when integers and floats had to be converted to temporary strings above.
 1082                     // Binary clipboard is ignored because it's documented that except for certain features,
 1083                     // binary clipboard variables are seen only up to the first binary zero (mostly to
 1084                     // simplify the code).
 1085                     right_length = (right.symbol == SYM_VAR) ? right.var->LengthIgnoreBinaryClip() : _tcslen(right_string);
 1086                     if (sym_assign_var // Since "right" is being appended onto a variable ("left"), an optimization is possible.
 1087                         && sym_assign_var->AppendIfRoom(right_string, (VarSizeType)right_length)) // But only if the target variable has enough remaining capacity.
 1088                     {
 1089                         // AppendIfRoom() always fails for VAR_CLIPBOARD, so below won't execute for it (which is
 1090                         // good because don't want clipboard to stay as SYM_VAR after the assignment. This is
 1091                         // because it simplifies the code not to have to worry about VAR_CLIPBOARD in BIFs, etc.)
 1092                         this_token.var = sym_assign_var; // Make the result a variable rather than a normal operand so that its
 1093                         this_token.symbol = SYM_VAR;     // address can be taken, and it can be passed ByRef. e.g. &(x+=1)
 1094                         goto push_this_token; // Skip over all other sections such as subsequent checks of sym_assign_var because it was all taken care of here.
 1095                     }
 1096                     // Otherwise, fall back to the other concat methods:
 1097                     left_length = (left.symbol == SYM_VAR) ? left.var->LengthIgnoreBinaryClip() : _tcslen(left_string);
 1098                     result_size = right_length + left_length + 1;
 1099 
 1100                     if (sym_assign_var)  // Fix for v1.0.48: These 2 lines were added, and they must take
 1101                         temp_var = NULL; // precendence over the other checks below to allow an expression like the following to work: var := var2 .= "abc"
 1102                     else if (output_var && EXPR_IS_DONE) // i.e. this is ACT_ASSIGNEXPR and we're at the final operator, a concat.
 1103                     {
 1104                         temp_var = output_var;
 1105                         done_and_have_an_output_var = TRUE;
 1106                     }
 1107                     else if (this_postfix[1].symbol == SYM_ASSIGN // Next operation is ":=".
 1108                         && stack_count && stack[stack_count-1]->symbol == SYM_VAR // i.e. let the next iteration handle it instead of doing it here.  Further below relies on this having been checked.
 1109                         && stack[stack_count-1]->var->Type() == VAR_NORMAL) // Don't do clipboard here because: 1) AcceptNewMem() doesn't support it; 2) Could probably use Assign() and then make its result be a newly added mem_count item, but the code complexity doesn't seem worth it given the rarity.
 1110                     {
 1111                         temp_var = stack[stack_count-1]->var;
 1112                         done_and_have_an_output_var = FALSE;
 1113                     }
 1114                     else
 1115                         temp_var = NULL;
 1116 
 1117                     if (temp_var)
 1118                     {
 1119                         result = temp_var->Contents(FALSE); // No need to update the contents because we just want to know if the current address of mContents matches some other addresses.
 1120                         if (result == Var::sEmptyString) // Added in v1.1.09.03.
 1121                         {
 1122                             // One of the following is true:
 1123                             //   1) temp_var has zero capacity and is empty.
 1124                             //   2) temp_var has zero capacity and contains an unflushed binary number.
 1125                             // In the first case, AppendIfRoom() will always fail, so we want to skip it and use
 1126                             // the "no overlap" optimization below. In the second case, calling AppendIfRoom()
 1127                             // would produce the wrong result; e.g. (x := 0+1, x := y 0) would produce "10".
 1128                             result = NULL;
 1129                         }
 1130                         if (result == left_string) // This is something like x := x . y, so simplify it to x .= y
 1131                         {
 1132                             // MUST DO THE ABOVE CHECK because the next section further below might free the
 1133                             // destination memory before doing the operation. Thus, if the destination is the
 1134                             // same as one of the sources, freeing it beforehand would obviously be a problem.
 1135                             if (temp_var->AppendIfRoom(right_string, (VarSizeType)right_length))
 1136                             {
 1137                                 if (done_and_have_an_output_var) // Fix for v1.0.48: Checking "temp_var == output_var" would not be enough for cases like v := (v := v . "a") . "b"
 1138                                     goto normal_end_skip_output_var; // Nothing more to do because it has even taken care of output_var already.
 1139                                 else // temp_var is from look-ahead to a future assignment.
 1140                                 {
 1141                                     this_token.circuit_token = (++this_postfix)->circuit_token; // Old, somewhat obsolete comment: this_postfix.circuit_token should have been NULL prior to this because the final right-side result of an assignment shouldn't be the last item of an AND/OR/IFF's left branch. The assignment itself would be that.
 1142                                     this_token.var = STACK_POP->var; // Make the result a variable rather than a normal operand so that its
 1143                                     this_token.symbol = SYM_VAR;     // address can be taken, and it can be passed ByRef. e.g. &(x:=1)
 1144                                     goto push_this_token;
 1145                                 }
 1146                             }
 1147                             //else no optimizations are possible because: 1) No room; 2) The overlap between the
 1148                             // source and dest requires temporary memory.  So fall through to the slower method.
 1149                         }
 1150                         else if (result != right_string) // No overlap between the two sources and dest.
 1151                         {
 1152                             // The check above assumes that only a complete equality/overlap is possible,
 1153                             // not a partial overlap.  A partial overlap between the memory of two variables
 1154                             // seems impossible for a script to produce.  But if it ever does happen, the
 1155                             // Assign() below would free part or all of one of the sources before doing
 1156                             // the concat, which would corrupt the result.
 1157                             // Optimize by copying directly into the target variable rather than the intermediate
 1158                             // step of putting into temporary memory.
 1159                             if (!temp_var->AssignString(NULL, (VarSizeType)result_size - 1)) // Resize the destination, if necessary.
 1160                                 goto abort; // Above should have already reported the error.
 1161                             result = temp_var->Contents(); // Call Contents() AGAIN because Assign() may have changed it.  No need to pass FALSE because the call to Assign() above already reset the contents.
 1162                             if (left_length)
 1163                                 tmemcpy(result, left_string, left_length);  // Not +1 because don't need the zero terminator.
 1164                             tmemcpy(result + left_length, right_string, right_length + 1); // +1 to include its zero terminator.
 1165                             temp_var->Close(); // Must be called after Assign(NULL, ...) or when Contents() has been altered because it updates the variable's attributes and properly handles VAR_CLIPBOARD.
 1166                             if (done_and_have_an_output_var) // Fix for v1.0.48: Checking "temp_var == output_var" would not be enough for cases like v := (v := "a" . "b") . "c".
 1167                                 goto normal_end_skip_output_var; // Nothing more to do because it has even taken care of output_var already.
 1168                             else // temp_var is from look-ahead to a future assignment.
 1169                             {
 1170                                 this_token.circuit_token = (++this_postfix)->circuit_token; // Old, somewhat obsolete comment: this_token.circuit_token should have been NULL prior to this because the final right-side result of an assignment shouldn't be the last item of an AND/OR/IFF's left branch. The assignment itself would be that.
 1171                                 this_token.var = STACK_POP->var; // Make the result a variable rather than a normal operand so that its
 1172                                 this_token.symbol = SYM_VAR;     // address can be taken, and it can be passed ByRef. e.g. &(x:=1)
 1173                                 goto push_this_token;
 1174                             }
 1175                         }
 1176                         //else result==right_string (e.g. x := y . x).  Although this could be optimized by 
 1177                         // moving memory around inside output_var (if it has enough capacity), it seems more
 1178                         // complicated than it's worth given the rarity of this.  It probably wouldn't save
 1179                         // much time anyway due to the memory-moves inside output_var.  So just fall through
 1180                         // to the normal method.
 1181                     } // if (temp_var)
 1182 
 1183                     // Since above didn't "goto", it didn't find a way to optimize this concat.
 1184                     // So fall back to the standard method.
 1185                     // The following section is similar to the one for "symbol == SYM_FUNC", so they
 1186                     // should be maintained together.
 1187                     // The following isn't done because there's a memcpy() further below which would also
 1188                     // have to check it, which hurts maintainability.  This doesn't seem worth it since
 1189                     // it's unlikely to be the empty string in the case of concat.
 1190                     //if (result_size == 1)
 1191                     //  this_token.marker = "";
 1192                     //else
 1193                     // Must cast to int to avoid loss of negative values:
 1194                     if (result_size <= (int)(aDerefBufSize - (target - aDerefBuf))) // There is room at the end of our deref buf, so use it.
 1195                     {
 1196                         this_token.marker = target;
 1197                         target += result_size;  // Adjust target for potential future use by another concat or function call.
 1198                     }
 1199                     else if (result_size < EXPR_SMALL_MEM_LIMIT && alloca_usage < EXPR_ALLOCA_LIMIT) // See comments at EXPR_SMALL_MEM_LIMIT.
 1200                     {
 1201                         this_token.marker = (LPTSTR)talloca(result_size);
 1202                         alloca_usage += result_size; // This might put alloca_usage over the limit by as much as EXPR_SMALL_MEM_LIMIT, but that is fine because it's more of a guideline than a limit.
 1203                     }
 1204                     else // Need to create some new persistent memory for our temporary use.
 1205                     {
 1206                         // See the nearly identical section higher above for comments:
 1207                         if (mem_count == MAX_EXPR_MEM_ITEMS // No more slots left (should be nearly impossible).
 1208                             || !(this_token.marker = mem[mem_count] = tmalloc(result_size)))
 1209                         {
 1210                             LineError(ERR_OUTOFMEM);
 1211                             goto abort;
 1212                         }
 1213                         ++mem_count;
 1214                     }
 1215                     if (left_length)
 1216                         tmemcpy(this_token.marker, left_string, left_length);  // Not +1 because don't need the zero terminator.
 1217                     tmemcpy(this_token.marker + left_length, right_string, right_length + 1); // +1 to include its zero terminator.
 1218 
 1219                     // For this new concat operator introduced in v1.0.31, it seems best to treat the
 1220                     // result as a SYM_STRING if either operand is a SYM_STRING.  That way, when the
 1221                     // result of the operation is later used, it will be a real string even if pure numeric,
 1222                     // which might affect the behavior of some things such as "casting" a string to a boolean,
 1223                     // e.g. if ("0" . 0)
 1224                     if (left.symbol == SYM_STRING || right.symbol == SYM_STRING)
 1225                         result_symbol = SYM_STRING;
 1226                     else
 1227                     {
 1228                         result_symbol = SYM_OPERAND;
 1229                         this_token.buf = NULL; // Indicate that this SYM_OPERAND token LACKS a pre-converted binary integer.
 1230                     }
 1231                     break;
 1232 
 1233                 default:
 1234                     // Other operators do not support string operands, so the result is an empty string.
 1235                     this_token.marker = _T("");
 1236                     result_symbol = SYM_STRING;
 1237                 }
 1238                 this_token.symbol = result_symbol; // Must be done only after the switch() above.
 1239             }
 1240 
 1241             else if (right_is_number == PURE_INTEGER && left_is_number == PURE_INTEGER && this_token.symbol != SYM_DIVIDE
 1242                 || this_token.symbol <= SYM_BITSHIFTRIGHT && this_token.symbol >= SYM_BITOR) // Check upper bound first for short-circuit performance (because operators like +-*/ are much more frequently used).
 1243             {
 1244                 // Because both are integers and the operation isn't division, the result is integer.
 1245                 // The result is also an integer for the bitwise operations listed in the if-statement
 1246                 // above.  This is because it is not legal to perform ~, &, |, or ^ on doubles, and also
 1247                 // because this behavior conforms to that of the Transform command.  Any floating point
 1248                 // operands are truncated to integers prior to doing the bitwise operation.
 1249                 right_int64 = TokenToInt64(right, right_is_number==PURE_INTEGER); // It can't be SYM_STRING because in here, both right and
 1250                 left_int64 = TokenToInt64(left, left_is_number==PURE_INTEGER);    // left are known to be numbers (otherwise an earlier "else if" would have executed instead of this one).
 1251                 result_symbol = SYM_INTEGER; // Set default.
 1252                 switch(this_token.symbol)
 1253                 {
 1254                 // The most common cases are kept up top to enhance performance if switch() is implemented as if-else ladder.
 1255                 case SYM_ADD:      this_token.value_int64 = left_int64 + right_int64; break;
 1256                 case SYM_SUBTRACT: this_token.value_int64 = left_int64 - right_int64; break;
 1257                 case SYM_MULTIPLY: this_token.value_int64 = left_int64 * right_int64; break;
 1258                 // A look at K&R confirms that relational/comparison operations and logical-AND/OR/NOT
 1259                 // always yield a one or a zero rather than arbitrary non-zero values:
 1260                 case SYM_EQUALCASE: // Same behavior as SYM_EQUAL for numeric operands.
 1261                 case SYM_EQUAL:    this_token.value_int64 = left_int64 == right_int64; break;
 1262                 case SYM_NOTEQUAL: this_token.value_int64 = left_int64 != right_int64; break;
 1263                 case SYM_GT:       this_token.value_int64 = left_int64 > right_int64; break;
 1264                 case SYM_LT:       this_token.value_int64 = left_int64 < right_int64; break;
 1265                 case SYM_GTOE:     this_token.value_int64 = left_int64 >= right_int64; break;
 1266                 case SYM_LTOE:     this_token.value_int64 = left_int64 <= right_int64; break;
 1267                 case SYM_BITAND:   this_token.value_int64 = left_int64 & right_int64; break;
 1268                 case SYM_BITOR:    this_token.value_int64 = left_int64 | right_int64; break;
 1269                 case SYM_BITXOR:   this_token.value_int64 = left_int64 ^ right_int64; break;
 1270                 case SYM_BITSHIFTLEFT:  this_token.value_int64 = left_int64 << right_int64; break;
 1271                 case SYM_BITSHIFTRIGHT: this_token.value_int64 = left_int64 >> right_int64; break;
 1272                 case SYM_FLOORDIVIDE:
 1273                     // Since it's integer division, no need for explicit floor() of the result.
 1274                     // Also, performance is much higher for integer vs. float division, which is part
 1275                     // of the justification for a separate operator.
 1276                     if (right_int64 == 0) // Divide by zero produces blank result (perhaps will produce exception if scripts ever support exception handlers).
 1277                     {
 1278                         this_token.marker = _T("");
 1279                         result_symbol = SYM_STRING;
 1280                     }
 1281                     else
 1282                         this_token.value_int64 = left_int64 / right_int64;
 1283                     break;
 1284                 case SYM_POWER:
 1285                     // Note: The function pow() in math.h adds about 28 KB of code size (uncompressed)!
 1286                     // Even assuming pow() supports negative bases such as (-2)**2, its size is why it's not used.
 1287                     // v1.0.44.11: With Laszlo's help, negative integer bases are now supported.
 1288                     if (!left_int64 && right_int64 < 0) // In essence, this is divide-by-zero.
 1289                     {
 1290                         // Return a consistent result rather than something that varies:
 1291                         this_token.marker = _T("");
 1292                         result_symbol = SYM_STRING;
 1293                     }
 1294                     else // We have a valid base and exponent and both are integers, so the calculation will always have a defined result.
 1295                     {
 1296                         if (left_was_negative = (left_int64 < 0))
 1297                             left_int64 = -left_int64; // Force a positive due to the limitations of qmathPow().
 1298                         this_token.value_double = qmathPow((double)left_int64, (double)right_int64);
 1299                         if (left_was_negative && right_int64 % 2) // Negative base and odd exponent (not zero or even).
 1300                             this_token.value_double = -this_token.value_double;
 1301                         if (right_int64 < 0)
 1302                             result_symbol = SYM_FLOAT; // Due to negative exponent, override to float like TRANS_CMD_POW.
 1303                         else
 1304                             this_token.value_int64 = (__int64)this_token.value_double;
 1305                     }
 1306                     break;
 1307                 }
 1308                 this_token.symbol = result_symbol; // Must be done only after the switch() above.
 1309             }
 1310 
 1311             else // Since one or both operands are floating point (or this is the division of two integers), the result will be floating point.
 1312             {
 1313                 right_double = TokenToDouble(right, TRUE, right_is_number==PURE_FLOAT); // Pass TRUE for aCheckForHex one of them is an integer to
 1314                 left_double = TokenToDouble(left, TRUE, left_is_number==PURE_FLOAT);   // be converted to a float for the purpose of this calculation.
 1315                 result_symbol = IS_RELATIONAL_OPERATOR(this_token.symbol) ? SYM_INTEGER : SYM_FLOAT; // Set default. v1.0.47.01: Changed relational operators to yield integers vs. floats because it's more intuitive and traditional (might also make relational operators perform better).
 1316                 switch(this_token.symbol)
 1317                 {
 1318                 case SYM_ADD:      this_token.value_double = left_double + right_double; break;
 1319                 case SYM_SUBTRACT: this_token.value_double = left_double - right_double; break;
 1320                 case SYM_MULTIPLY: this_token.value_double = left_double * right_double; break;
 1321                 case SYM_DIVIDE:
 1322                 case SYM_FLOORDIVIDE:
 1323                     if (right_double == 0.0) // Divide by zero produces blank result.
 1324                     {
 1325                         this_token.marker = _T("");
 1326                         result_symbol = SYM_STRING;
 1327                     }
 1328                     else
 1329                     {
 1330                         this_token.value_double = left_double / right_double;
 1331                         if (this_token.symbol == SYM_FLOORDIVIDE) // Like Python, the result is floor()'d, moving to the nearest integer to the left on the number line.
 1332                             this_token.value_double = qmathFloor(this_token.value_double); // Result is always a double when at least one of the inputs was a double.
 1333                     }
 1334                     break;
 1335                 case SYM_EQUALCASE: // Same behavior as SYM_EQUAL for numeric operands.
 1336                 case SYM_EQUAL:    this_token.value_int64 = left_double == right_double; break;
 1337                 case SYM_NOTEQUAL: this_token.value_int64 = left_double != right_double; break;
 1338                 case SYM_GT:       this_token.value_int64 = left_double > right_double; break;
 1339                 case SYM_LT:       this_token.value_int64 = left_double < right_double; break;
 1340                 case SYM_GTOE:     this_token.value_int64 = left_double >= right_double; break;
 1341                 case SYM_LTOE:     this_token.value_int64 = left_double <= right_double; break;
 1342                 case SYM_POWER:
 1343                     // v1.0.44.11: With Laszlo's help, negative bases are now supported as long as the exponent is not fractional.
 1344                     // See the other SYM_POWER higher above for more details about below.
 1345                     left_was_negative = (left_double < 0);
 1346                     if (left_double == 0.0 && right_double < 0  // In essence, this is divide-by-zero.
 1347                         || left_was_negative && qmathFmod(right_double, 1.0) != 0.0) // Negative base, but exponent isn't close enough to being an integer: unsupported (to simplify code).
 1348                     {
 1349                         this_token.marker = _T("");
 1350                         result_symbol = SYM_STRING;
 1351                     }
 1352                     else
 1353                     {
 1354                         if (left_was_negative)
 1355                             left_double = -left_double; // Force a positive due to the limitations of qmathPow().
 1356                         this_token.value_double = qmathPow(left_double, right_double);
 1357                         if (left_was_negative && qmathFabs(qmathFmod(right_double, 2.0)) == 1.0) // Negative base and exactly-odd exponent (otherwise, it can only be zero or even because if not it would have returned higher above).
 1358                             this_token.value_double = -this_token.value_double;
 1359                     }
 1360                     break;
 1361                 } // switch(this_token.symbol)
 1362                 this_token.symbol = result_symbol; // Must be done only after the switch() above.
 1363             } // Result is floating point.
 1364         } // switch() operator type
 1365 
 1366         if (sym_assign_var) // Added in v1.0.46. There are some places higher above that handle sym_assign_var themselves and skip this section via goto.
 1367         {
 1368             if (!sym_assign_var->Assign(this_token)) // Assign the result (based on its type) to the target variable.
 1369                 goto abort;
 1370             if (sym_assign_var->Type() != VAR_CLIPBOARD)
 1371             {
 1372                 this_token.var = sym_assign_var;    // Make the result a variable rather than a normal operand so that its
 1373                 this_token.symbol = SYM_VAR;        // address can be taken, and it can be passed ByRef. e.g. &(x+=1)
 1374             }
 1375             //else its the clipboard, so just push this_token as-is because after its assignment is done,
 1376             // VAR_CLIPBOARD should no longer be a SYM_VAR.  This is done to simplify the code, such as BIFs.
 1377             //
 1378             // Now fall through and push this_token onto the stack as an operand for use by future operators.
 1379             // This is because by convention, an assignment like "x+=1" produces a usable operand.
 1380         }
 1381 
 1382 push_this_token:
 1383         if (!this_token.circuit_token) // It's not capable of short-circuit.
 1384         {
 1385             while (high_water_mark > stack_count)
 1386             {   // L31: Release any objects which have been previously popped off the stack. This seems
 1387                 // to be the simplest way to do it as tokens are popped off the stack at multiple points,
 1388                 // but only this one point where parameters are pushed.  high_water_mark allows us to determine
 1389                 // if there were tokens on the stack before returning, regardless of how expression evaluation
 1390                 // ended (abort, abnormal_end, normal_end_skip_output_var...).  This method also ensures any
 1391                 // objects passed as parameters to a function (such as ObjGet()) are released *AFTER* the return
 1392                 // value is made persistent, which is important if the return value refers to the object's memory.
 1393                 --high_water_mark;
 1394                 if (stack[high_water_mark]->symbol == SYM_OBJECT)
 1395                     stack[high_water_mark]->object->Release();
 1396             }
 1397             STACK_PUSH(&this_token);   // Push the result onto the stack for use as an operand by a future operator.
 1398             high_water_mark = stack_count; // L31
 1399         }
 1400         else // This is the final result of an IFF's condition or a AND or OR's left branch.  Apply short-circuit boolean method to it.
 1401         {
 1402 non_null_circuit_token:
 1403             // Cast this left-branch result to true/false, then determine whether it should cause its
 1404             // parent AND/OR/IFF to short-circuit.
 1405             left_branch_is_true = TokenToBOOL(this_token);
 1406             if (this_token.symbol == SYM_OBJECT)
 1407                 this_token.object->Release();
 1408             if (this_token.circuit_token->symbol == SYM_IFF_THEN)
 1409             {
 1410                 if (!left_branch_is_true) // The ternary's condition is false.
 1411                 {
 1412                     // Discard the entire "then" branch of this ternary operator, leaving only the
 1413                     // "else" branch to be evaluated later as the result.
 1414                     // Ternaries nested inside each other don't need to be considered special for the purpose
 1415                     // of discarding ternary branches due to the very nature of postfix (i.e. it's already put
 1416                     // nesting in the right postfix order to support this method of discarding a branch).
 1417                     this_postfix = this_token.circuit_token; // The address in any circuit_token always points into the arg's postfix array (never any temporary array or token created here) due to the nature/definition of circuit_token.
 1418                     // The outer loop will now discard the SYM_IFF_THEN itself.
 1419                 }
 1420                 //else the ternary's condition is true.  Do nothing; just let the next iteration evaluate the
 1421                 // THEN portion and then treat the SYM_IFF_THEN it encounters as a unary operator (after that,
 1422                 // it will discard the ELSE branch).
 1423                 continue;
 1424             }
 1425             // Since above didn't "continue", this_token is the left branch of an AND/OR.  Check for short-circuit.
 1426             // The following loop exists to support cascading short-circuiting such as the following example:
 1427             // 2>3 and 2>3 and 2>3
 1428             // In postfix notation, the above looks like:
 1429             // 2 3 > 2 3 > and 2 3 > and
 1430             // When the first '>' operator is evaluated to false, it sees that its parent is an AND and
 1431             // thus it short-circuits, discarding everything between the first '>' and the "and".
 1432             // But since the first and's parent is the second "and", that false result just produced is now
 1433             // the left branch of the second "and", so the loop conducts a second iteration to discard
 1434             // everything between the first "and" and the second.  By contrast, if the second "and" were
 1435             // an "or", the second iteration would never occur because the loop's condition would be false
 1436             // on the second iteration, which would then cause the first and's false value to be discarded
 1437             // (due to the loop ending without having PUSHed) because solely the right side of the "or" should
 1438             // determine the final result of the "or".
 1439             //
 1440             // The following code is probably equivalent to the loop below it.  However, it's only slightly
 1441             // smaller in code size when you examine what it actually does, and it almost certainly performs
 1442             // slightly worse because the "goto" incurs unnecessary steps such as recalculating left_branch_is_true.
 1443             // Therefore, it doesn't seem worth changing it:
 1444             //if (left_branch_is_true == (this_token.circuit_token->symbol == SYM_OR)) // If true, this AND/OR causes a short-circuit
 1445             //{
 1446             //  for (++i; postfix+i != this_token.circuit_token; ++i); // (This line obsolete; needs revision.) Should always be found, so no need to guard against reading beyond the end of the array.
 1447             //  this_token.symbol = SYM_INTEGER;
 1448             //  this_token.value_int64 = left_branch_is_true; // Assign a pure 1 (for SYM_OR) or 0 (for SYM_AND).
 1449             //  this_token.circuit_token = this_postfix->circuit_token; // In case circuit_token == SYM_IFF_THEN.
 1450             //  goto push_this_token; // In lieu of STACK_PUSH(this_token) in case circuit_token == SYM_IFF_THEN.
 1451             //}
 1452             for (circuit_token = this_token.circuit_token
 1453                 ; left_branch_is_true == (circuit_token->symbol == SYM_OR);) // If true, this AND/OR causes a short-circuit
 1454             {
 1455                 // Discard the entire right branch of this AND/OR:
 1456                 this_postfix = circuit_token; // The address in any circuit_token always points into the arg's postfix array (never any temporary array or token created here) due to the nature/definition of circuit_token.
 1457                 if (   !(circuit_token = this_postfix->circuit_token) // This value is also used by our loop's condition. Relies on short-circuit boolean order with the below.
 1458                     || circuit_token->symbol == SYM_IFF_THEN   ) // Don't cascade from AND/OR into IFF because IFF requires a different cascade approach that's implemented only after its winning branch is evaluated.  Otherwise, things like "0 and 1 ? 3 : 4" wouldn't work.
 1459                 {
 1460                     // No more cascading is needed because this AND/OR isn't the left branch of another.
 1461                     // This will be the final result of this AND/OR because it's right branch was discarded
 1462                     // above without having been evaluated nor any of its functions called.
 1463                     this_token.symbol = SYM_INTEGER;
 1464                     this_token.value_int64 = left_branch_is_true; // Assign a pure 1 (for SYM_OR) or 0 (for SYM_AND).
 1465                     this_token.circuit_token = circuit_token; // In case circuit_token->symbol == SYM_IFF_THEN.
 1466                     goto push_this_token; // In lieu of STACK_PUSH(this_token) in case circuit_token->symbol == SYM_IFF_THEN.
 1467                 }
 1468                 //else there is more cascading to be checked, so continue looping.
 1469             }
 1470             // If the loop ends normally (not via "break"), this_postfix is now the left branch of an
 1471             // AND/OR that should not short-circuit.  As a result, this left branch is simply discarded
 1472             // (by means of the outer loop) because its right branch will be the sole determination
 1473             // of whether this AND/OR is true or false.
 1474         } // Short-circuit (an IFF or the left branch of an AND/OR).
 1475     } // For each item in the postfix array.
 1476 
 1477     // Although ACT_EXPRESSION was already checked higher above for function calls, there are other ways besides
 1478     // an isolated function call to have ACT_EXPRESSION.  For example: var&=3 (where &= is an operator that lacks
 1479     // a corresponding command).  Another example: true ? fn1() : fn2()
 1480     // Also, there might be ways the function-call section didn't return for ACT_EXPRESSION, such as when somehow
 1481     // there was more than one token on the stack even for the final function call, or maybe other unforeseen ways.
 1482     // It seems best to avoid any chance of looking at the result since it might be invalid due to the above
 1483     // having taken shortcuts (since it knew the result would be discarded).
 1484     if (mActionType == ACT_EXPRESSION)   // A stand-alone expression whose end result doesn't matter.
 1485         goto normal_end_skip_output_var; // Can't be any output_var for this action type. Also, leave result_to_return at its default of "".
 1486 
 1487     if (stack_count != 1)  // Even for multi-statement expressions, the stack should have only one item left on it:
 1488         goto abnormal_end; // the overall result. Examples of errors include: () ... x y ... (x + y) (x + z) ... etc. (some of these might no longer produce this issue due to auto-concat).
 1489 
 1490     ExprTokenType &result_token = *stack[0];  // For performance and convenience.  Even for multi-statement, the bottommost item on the stack is the final result so that things like var1:=1,var2:=2 work.
 1491 
 1492     // Store the result of the expression in the deref buffer for the caller.  It is stored in the current
 1493     // format in effect via SetFormat because:
 1494     // 1) The := operator then doesn't have to convert to int/double then back to string to put the right format into effect.
 1495     // 2) It might add a little bit of flexibility in places parameters where floating point values are expected
 1496     //    (i.e. it allows a way to do automatic rounding), without giving up too much.  Changing floating point
 1497     //    precision from the default of 6 decimal places is rare anyway, so as long as this behavior is documented,
 1498     //    it seems okay for the moment.
 1499     if (output_var)
 1500     {
 1501         // v1.0.45: Take a shortcut, which in the case of SYM_STRING/OPERAND/VAR avoids one memcpy
 1502         // (into the deref buffer).  In some cases, this also saves from having to expand the deref buffer.
 1503         if (!output_var->Assign(result_token))
 1504             goto abort;
 1505         goto normal_end_skip_output_var; // result_to_return is left at its default of "", though its value doesn't matter as long as it isn't NULL.
 1506     }
 1507 
 1508     if (mActionType == ACT_IFEXPR || mActionType == ACT_WHILE || mActionType == ACT_UNTIL) // This is an optimization that improves the speed of ACT_IFEXPR by up to 50% (ACT_WHILE is probably improved by only up-to-15%). Simple expressions like "if (x < y)" see the biggest speedup.
 1509     {
 1510         BOOL result_is_true;
 1511         switch (result_token.symbol)
 1512         {
 1513         case SYM_INTEGER: result_is_true = (result_token.value_int64 != 0); break;
 1514         case SYM_FLOAT:   result_is_true = (result_token.value_double != 0.0); break;
 1515         case SYM_OPERAND:
 1516             if (result_token.buf)
 1517             {
 1518                 result_is_true = (*(__int64 *)result_token.buf != 0); // Use the stored binary integer for performance.
 1519                 break;
 1520             }
 1521             //else DON'T BREAK; FALL THROUGH TO NEXT CASE:
 1522         case SYM_STRING: // *** OR IT FELL THROUGH FROM ABOVE CASE ***
 1523             result_is_true = LegacyResultToBOOL(result_token.marker);
 1524             break;
 1525         case SYM_VAR: // SYM_VAR is somewhat unusual at this late a stage.
 1526             result_is_true = LegacyVarToBOOL(*result_token.var);
 1527             break;
 1528         case SYM_OBJECT: // L31: Objects are always treated as TRUE values.
 1529             result_is_true = true;
 1530             break;
 1531         }
 1532         result_to_return = result_is_true ? _T("1") : _T(""); // Return "" vs. "0" for FALSE for consistency with "goto abnormal_end" (which bypasses this section).
 1533         goto normal_end_skip_output_var; // ACT_IFEXPR never has an output_var.
 1534     }
 1535 
 1536     if (aResultToken) // L31
 1537     {
 1538         if (IS_NUMERIC(result_token.symbol) || result_token.symbol == SYM_OBJECT)
 1539         {   // Return numeric or object result as-is.
 1540             aResultToken->symbol = result_token.symbol;
 1541             aResultToken->value_int64 = result_token.value_int64;
 1542             return _T(""); // Must not return NULL; any other value is OK (will be ignored).
 1543         }
 1544         if (result_token.symbol == SYM_VAR && result_token.var->HasObject())
 1545         {   // L34: Allow returning of objects contained by variables; 'return var' was already supported since that is not treated as an expression.
 1546             aResultToken->symbol = SYM_OBJECT;
 1547             aResultToken->object = result_token.var->Object();
 1548             aResultToken->object->AddRef();
 1549             return _T("");
 1550         }
 1551     }
 1552     //else result is a string.  Since it may be contained by a temporary memory block which we will free before returning, just return it as per usual.
 1553 
 1554     // Otherwise:
 1555     result_to_return = aTarget; // Set default.
 1556     switch (result_token.symbol)
 1557     {
 1558     case SYM_INTEGER:
 1559         // SYM_INTEGER and SYM_FLOAT will fit into our deref buffer because an earlier stage has already ensured
 1560         // that the buffer is large enough to hold at least one number.  But a string/generic might not fit if it's
 1561         // a concatenation and/or a large string returned from a called function.
 1562         aTarget += _tcslen(ITOA64(result_token.value_int64, aTarget)) + 1; // Store in hex or decimal format, as appropriate.
 1563         // Above: +1 because that's what callers want; i.e. the position after the terminator.
 1564         goto normal_end_skip_output_var; // output_var was already checked higher above, so no need to consider it again.
 1565     case SYM_FLOAT:
 1566         // In case of float formats that are too long to be supported, use snprint() to restrict the length.
 1567          // %f probably defaults to %0.6f.  %f can handle doubles in MSVC++.
 1568         aTarget += sntprintf(aTarget, MAX_NUMBER_SIZE, g->FormatFloat, result_token.value_double) + 1; // +1 because that's what callers want; i.e. the position after the terminator.
 1569         goto normal_end_skip_output_var; // output_var was already checked higher above, so no need to consider it again.
 1570     case SYM_STRING:
 1571     case SYM_OPERAND:
 1572     case SYM_VAR: // SYM_VAR is somewhat unusual at this late a stage.
 1573         // At this stage, we know the result has to go into our deref buffer because if a way existed to
 1574         // avoid that, we would already have goto/returned higher above (e.g. for ACT_ASSIGNEXPR OR ACT_EXPRESSION.
 1575         // Also, at this stage, the pending result can exist in one of several places:
 1576         // 1) Our deref buf (due to being a single-deref, a function's return value that was copied to the
 1577         //    end of our buf because there was enough room, etc.)
 1578         // 2) In a called function's deref buffer, namely sDerefBuf, which will be deleted by our caller
 1579         //    shortly after we return to it.
 1580         // 3) In an area of memory we alloc'd for lack of any better place to put it.
 1581         if (result_token.symbol == SYM_VAR)
 1582         {
 1583             result = result_token.var->Contents();
 1584             result_size = result_token.var->LengthIgnoreBinaryClip() + 1; // Ignore binary clipboard for anything other than ACT_ASSIGNEXPR (i.e. output_var!=NULL) because it's documented that except for certain features, binary clipboard variables are seen only up to the first binary zero (mostly to simplify the code).
 1585         }
 1586         else
 1587         {
 1588             result = result_token.marker;
 1589             result_size = _tcslen(result) + 1;
 1590         }
 1591 
 1592         // Notes about the macro below:
 1593         // Space is needed for whichever of the following is greater (since only one of the following is in
 1594         // the deref buf at any given time; i.e. they can share the space by being in it at different times):
 1595         // 1) All the expression's literal strings/numbers and double-derefs (e.g. "Array%i%" as a string).
 1596         //    Allowing room for this_arg.length plus a terminator seems enough for any conceivable
 1597         //    expression, even worst-cases and malformatted syntax-error expressions. This is because
 1598         //    every numeric literal or double-deref needs to have some kind of symbol or character
 1599         //    between it and the next one or it would never have been recognized as a separate operand
 1600         //    in the first place.  And the final item uses the final terminator provided via +1 below.
 1601         // 2) Any numeric result (i.e. MAX_NUMBER_LENGTH).  If the expression needs to store a string
 1602         //    result, it will take care of expanding the deref buffer.
 1603         #define EXPR_BUF_SIZE(raw_expr_len) (raw_expr_len < MAX_NUMBER_LENGTH \
 1604             ? MAX_NUMBER_LENGTH : raw_expr_len) + 1 // +1 for the overall terminator.
 1605 
 1606         // If result is the empty string or a number, it should always fit because the size estimation
 1607         // phase has ensured that capacity_of_our_buf_portion is large enough to hold those.
 1608         // In addition, it doesn't matter if we already used target/aTarget for things higher above
 1609         // because anything in there we're now done with, and memmove() vs. memcpy() later below
 1610         // will allow overlap of the final result with intermediate results already in the buffer.
 1611         size_t capacity_of_our_buf_portion;
 1612         capacity_of_our_buf_portion = EXPR_BUF_SIZE(mArg[aArgIndex].length) + aExtraSize; // The initial amount of size available to write our final result.
 1613         if (result_size > capacity_of_our_buf_portion)
 1614         {
 1615             // Do a simple expansion of our deref buffer to handle the fact that our actual result is bigger
 1616             // than the size estimator could have calculated (due to a concatenation or a large string returned
 1617             // from a called function).  This performs poorly but seems justified by the fact that it typically
 1618             // happens only in extreme cases.
 1619             size_t new_buf_size = aDerefBufSize + (result_size - capacity_of_our_buf_portion);
 1620 
 1621             // malloc() and free() are used instead of realloc() because in many cases, the overhead of
 1622             // realloc()'s internal memcpy(entire contents) can be avoided because only part or
 1623             // none of the contents needs to be copied (realloc's ability to do an in-place resize might
 1624             // be unlikely for anything other than small blocks; see compiler's realloc.c):
 1625             LPTSTR new_buf;
 1626             if (   !(new_buf = tmalloc(new_buf_size))   )
 1627             {
 1628                 LineError(ERR_OUTOFMEM);
 1629                 goto abort;
 1630             }
 1631             if (new_buf_size > LARGE_DEREF_BUF_SIZE)
 1632                 ++sLargeDerefBufs; // And if the old deref buf was larger too, this value is decremented later below. SET_DEREF_TIMER() is handled by our caller because aDerefBufSize is updated further below, which the caller will see.
 1633 
 1634             // Copy only that portion of the old buffer that is in front of our portion of the buffer
 1635             // because we no longer need our portion (except for result.marker if it happens to be
 1636             // in the old buffer, but that is handled after this):
 1637             size_t aTarget_offset = aTarget - aDerefBuf;
 1638             if (aTarget_offset) // aDerefBuf has contents that must be preserved.
 1639                 tmemcpy(new_buf, aDerefBuf, aTarget_offset); // This will also copy the empty string if the buffer first and only character is that.
 1640             aTarget = new_buf + aTarget_offset;
 1641             result_to_return = aTarget; // Update to reflect new value above.
 1642             // NOTE: result_token.marker might extend too far to the right in our deref buffer and thus be
 1643             // larger than capacity_of_our_buf_portion because other arg(s) exist in this line after ours
 1644             // that will be using a larger total portion of the buffer than ours.  Thus, the following must be
 1645             // done prior to free(), but memcpy() vs. memmove() is safe in any case:
 1646             tmemcpy(aTarget, result, result_size); // Copy from old location to the newly allocated one.
 1647 
 1648             free(aDerefBuf); // Free our original buffer since it's contents are no longer needed.
 1649             if (aDerefBufSize > LARGE_DEREF_BUF_SIZE)
 1650                 --sLargeDerefBufs;
 1651 
 1652             // Now that the buffer has been enlarged, need to adjust any other pointers that pointed into
 1653             // the old buffer:
 1654             LPTSTR aDerefBuf_end = aDerefBuf + aDerefBufSize; // Point it to the character after the end of the old buf.
 1655             for (i = 0; i < aArgIndex; ++i) // Adjust each item beneath ours (if any). Our own is not adjusted because we'll be returning the right address to our caller.
 1656                 if (aArgDeref[i] >= aDerefBuf && aArgDeref[i] < aDerefBuf_end)
 1657                     aArgDeref[i] = new_buf + (aArgDeref[i] - aDerefBuf); // Set for our caller.
 1658             // The following isn't done because target isn't used anymore at this late a stage:
 1659             //target = new_buf + (target - aDerefBuf);
 1660             aDerefBuf = new_buf; // Must be the last step, since the old address is used above.  Set for our caller.
 1661             aDerefBufSize = new_buf_size; // Set for our caller.
 1662         }
 1663         else // Deref buf is already large enough to fit the string.
 1664             if (aTarget != result) // Currently, might be always true.
 1665                 tmemmove(aTarget, result, result_size); // memmove() vs. memcpy() in this case, since source and dest might overlap (i.e. "target" may have been used to put temporary things into aTarget, but those things are no longer needed and now safe to overwrite).
 1666         aTarget += result_size;
 1667         goto normal_end_skip_output_var; // output_var was already checked higher above, so no need to consider it again.
 1668 
 1669     case SYM_OBJECT: // L31: Objects are always treated as empty strings; except with ACT_RETURN, which was handled above, and any usage which expects a boolean result.
 1670         result_to_return = _T("");
 1671         goto normal_end_skip_output_var;
 1672 
 1673     default: // Result contains a non-operand symbol such as an operator.
 1674         goto abnormal_end;
 1675     } // switch (result_token.symbol)
 1676 
 1677 // ALL PATHS ABOVE SHOULD "GOTO".  TO CATCH BUGS, ANY THAT DON'T FALL INTO "ABORT" BELOW.
 1678 abort:
 1679     // The callers of this function know that the value of aResult (which contains the reason
 1680     // for early exit) should be considered valid/meaningful only if result_to_return is NULL.
 1681     result_to_return = NULL; // Use NULL to inform our caller that this entire thread is to be terminated.
 1682     aResult = FAIL; // Indicate reason to caller.
 1683     goto normal_end_skip_output_var; // output_var is skipped as part of standard abort behavior.
 1684 
 1685 abnormal_end: // Currently the same as normal_end; it's separate to improve readability.  When this happens, result_to_return is typically "" (unless the caller overrode that default).
 1686 //normal_end: // This isn't currently used, but is available for future-use and readability.
 1687     // v1.0.45: ACT_ASSIGNEXPR relies on us to set the output_var (i.e. whenever it's ARG1's is_expression==true).
 1688     // Our taking charge of output_var allows certain performance optimizations in other parts of this function,
 1689     // such as avoiding excess memcpy's and malloc's during intermediate stages.
 1690     if (output_var && result_to_return) // i.e. don't assign if NULL to preserve backward compatibility with scripts that rely on the old value being changed in cases where an expression fails (unlikely).
 1691         if (!output_var->Assign(result_to_return))
 1692             aResult = FAIL;
 1693 
 1694 normal_end_skip_output_var:
 1695     for (i = mem_count; i--;) // Free any temporary memory blocks that were used.  Using reverse order might reduce memory fragmentation a little (depending on implementation of malloc).
 1696         free(mem[i]);
 1697 
 1698     // L31: Release any objects which have been previous pushed onto the stack and not yet released.
 1699     while (high_water_mark)
 1700     {   // See similar section under push_this_token for comments.
 1701         --high_water_mark;
 1702         if (stack[high_water_mark]->symbol == SYM_OBJECT)
 1703             stack[high_water_mark]->object->Release();
 1704     }
 1705 
 1706     return result_to_return;
 1707 
 1708     // Listing the following label last should slightly improve performance because it avoids an extra "goto"
 1709     // in the postfix loop above the push_this_token label.  Also, keeping seldom-reached code at the end
 1710     // may improve how well the code fits into the CPU cache.
 1711 double_deref_fail: // For the rare cases when the name of a dynamic function call is too long or improper.
 1712     // A deref with a NULL marker terminates the list, and also indicates whether this is a dynamic function
 1713     // call. "deref" has been set by the caller, and may or may not be the NULL marker deref.
 1714     for (; deref->marker; ++deref);
 1715     if (deref->is_function)
 1716         goto abnormal_end;
 1717     else
 1718         goto push_this_token;
 1719 }
 1720 
 1721 
 1722 
 1723 ResultType Line::ExpandSingleArg(int aArgIndex, ExprTokenType &aResultToken, LPTSTR &aDerefBuf, size_t &aDerefBufSize)
 1724 {
 1725     ExprTokenType *postfix = mArg[aArgIndex].postfix;
 1726     if (postfix->symbol < SYM_DYNAMIC // i.e. any other operand type.
 1727         && postfix->symbol != SYM_VAR // Variables must be dereferenced.
 1728         && postfix[1].symbol == SYM_INVALID) // Exactly one token.
 1729     {
 1730         aResultToken.symbol = postfix->symbol;
 1731         aResultToken.value_int64 = postfix->value_int64;
 1732 #ifdef _WIN64
 1733         aResultToken.buf = postfix->buf;
 1734 #endif
 1735         return OK;
 1736     }
 1737 
 1738     size_t space_needed = EXPR_BUF_SIZE(mArg[aArgIndex].length);
 1739 
 1740     if (aDerefBufSize < space_needed)
 1741     {
 1742         if (aDerefBuf)
 1743         {
 1744             free(aDerefBuf);
 1745             if (aDerefBufSize > LARGE_DEREF_BUF_SIZE)
 1746                 --sLargeDerefBufs;
 1747         }
 1748         if ( !(aDerefBuf = tmalloc(space_needed)) )
 1749         {
 1750             aDerefBufSize = 0;
 1751             return LineError(ERR_OUTOFMEM);
 1752         }
 1753         aDerefBufSize = space_needed;
 1754         if (aDerefBufSize > LARGE_DEREF_BUF_SIZE)
 1755             ++sLargeDerefBufs;
 1756     }
 1757 
 1758     // Use the whole buf:
 1759     LPTSTR buf_marker = aDerefBuf;
 1760     size_t extra_size = aDerefBufSize - space_needed;
 1761 
 1762     // None of the previous args (if any) are needed, so pass an array of NULLs:
 1763     LPTSTR arg_deref[MAX_ARGS];
 1764     for (int i = 0; i < aArgIndex; i++)
 1765         arg_deref[i] = NULL;
 1766 
 1767     // Initialize aResultToken so we can detect when ExpandExpression() uses it:
 1768     aResultToken.symbol = SYM_INVALID;
 1769     
 1770     ResultType result;
 1771     LPTSTR string_result = ExpandExpression(aArgIndex, result, &aResultToken, buf_marker, aDerefBuf, aDerefBufSize, arg_deref, extra_size);
 1772     if (!string_result)
 1773         return result; // Should be EARLY_EXIT or FAIL.
 1774 
 1775     if (aResultToken.symbol == SYM_INVALID) // It wasn't set by ExpandExpression().
 1776     {
 1777         aResultToken.symbol = SYM_STRING;
 1778         aResultToken.marker = string_result;
 1779     }
 1780     return OK;
 1781 }
 1782 
 1783 
 1784 
 1785 bool Func::Call(UDFCallInfo &aFuncCall, ResultType &aResult, ExprTokenType &aResultToken, ExprTokenType *aParam[], int aParamCount, bool aIsVariadic)
 1786 // aFuncCall: Caller passes a variable which should go out of scope after the function call's result
 1787 //   has been used; this automatically frees and restores a UDFs local vars (where applicable).
 1788 // aSpaceAvailable: -1 indicates this is a regular function call.  Otherwise this must be the amount of
 1789 //   space available after aParam for expanding the array of parameters for a variadic function call.
 1790 {
 1791     aResult = OK; // Set default.
 1792     
 1793     Object *param_obj = NULL;
 1794     if (aIsVariadic) // i.e. this is a variadic function call.
 1795     {
 1796         ExprTokenType *rvalue = NULL;
 1797         if (mName == (LPTSTR)IT_SET && aParamCount > 1) // x[y*]:=z
 1798             rvalue = aParam[--aParamCount];
 1799         
 1800         --aParamCount; // i.e. make aParamCount the count of normal params.
 1801         if (param_obj = dynamic_cast<Object *>(TokenToObject(*aParam[aParamCount])))
 1802         {
 1803             int extra_params = param_obj->MaxIndex();
 1804             if (extra_params > 0 || param_obj->HasNonnumericKeys())
 1805             {
 1806                 // Calculate space required for ...
 1807                 size_t space_needed = extra_params * sizeof(ExprTokenType) // ... new param tokens
 1808                     + max(mParamCount, aParamCount + extra_params) * sizeof(ExprTokenType *); // ... existing and new param pointers
 1809                 if (rvalue)
 1810                     space_needed += sizeof(rvalue); // ... extra slot for aRValue
 1811                 // Allocate new param list and tokens; tokens first for convenience.
 1812                 ExprTokenType *token = (ExprTokenType *)_alloca(space_needed);
 1813                 ExprTokenType **param_list = (ExprTokenType **)(token + extra_params);
 1814                 // Since built-in functions don't have variables we can directly assign to,
 1815                 // we need to expand the param object's contents into an array of tokens:
 1816                 param_obj->ArrayToParams(token, param_list, extra_params, aParam, aParamCount);
 1817                 aParam = param_list;
 1818                 aParamCount += extra_params;
 1819             }
 1820         }
 1821         if (rvalue)
 1822             aParam[aParamCount++] = rvalue; // In place of the variadic param.
 1823         // mMinParams isn't validated at load-time for variadic calls, so we must do it here:
 1824         // However, this check must be skipped for user-defined functions so that a named value
 1825         // can be supplied for a required parameter.  Missing required parameters are detected
 1826         // in the loop below by the absence of a default value.
 1827         if (aParamCount < mMinParams && mIsBuiltIn)
 1828             return false; // Abort expression.
 1829         // Otherwise, even if some params are SYM_MISSING, it is relatively safe to call the function.
 1830         // The TokenTo' set of functions will produce 0 or "" for missing params.  Although that isn't
 1831         // technically correct, it seems preferable over silently aborting the call.
 1832     }
 1833 
 1834     if (mIsBuiltIn)
 1835     {
 1836         aResultToken.symbol = SYM_INTEGER; // Set default return type so that functions don't have to do it if they return INTs.
 1837         aResultToken.marker = mName;       // Inform function of which built-in function called it (allows code sharing/reduction). Can't use circuit_token because it's value is still needed later below.
 1838 
 1839         // CALL THE BUILT-IN FUNCTION:
 1840         mBIF(aResult, aResultToken, aParam, aParamCount);
 1841         
 1842         if (g->ThrownToken)
 1843             aResult = FAIL; // Abort thread.
 1844     }
 1845     else // It's not a built-in function, or it's a built-in that was overridden with a custom function.
 1846     {
 1847         int j, count_of_actuals_that_have_formals;
 1848         count_of_actuals_that_have_formals = (aParamCount > mParamCount)
 1849             ? mParamCount  // Omit any actuals that lack formals (this can happen when a dynamic call passes too many parameters).
 1850             : aParamCount;
 1851         
 1852         // If there are other instances of this function already running, either via recursion or
 1853         // an interrupted quasi-thread, back up the local variables of the instance that lies immediately
 1854         // beneath ours (in turn, that instance is responsible for backing up any instance that lies
 1855         // beneath it, and so on, since when recursion collapses or threads resume, they always do so
 1856         // in the reverse order in which they were created.
 1857         //
 1858         // I think the backup-and-restore approach to local variables might enhance performance over
 1859         // other approaches, perhaps a lot.  This is because most of the time there will be no other
 1860         // instances of a given function on the call stack, thus no backup/restore is needed, and thus
 1861         // the function's existing local variables can be reused as though they're globals (i.e.
 1862         // memory allocation/deallocation overhead is often completely avoided for non-recursive calls
 1863         // to a function after the first).
 1864         if (mInstances > 0) // i.e. treat negatives as zero to help catch any bugs in the way mInstances is maintained.
 1865         {
 1866             // Backup/restore of function's variables is needed.
 1867             // Only when a backup is needed is it possible for this function to be calling itself recursively,
 1868             // either directly or indirectly by means of an intermediate function.  As a consequence, it's
 1869             // possible for this function to be passing one or more of its own params or locals to itself.
 1870             // The following section compensates for that to handle parameters passed by-value, but it
 1871             // doesn't correctly handle passing its own locals/params to itself ByRef, which is in the
 1872             // help file as a known limitation.  Also, the below doesn't indicate a failure when stack
 1873             // underflow would occur because the loop after this one needs to do that (since this
 1874             // one will never execute if a backup isn't needed).  Note that this loop that reviews all
 1875             // actual parameters is necessary as a separate loop from the one further below because this
 1876             // first one's conversion must occur prior to calling BackupFunctionVars().  In addition, there
 1877             // might be other interdependencies between formals and actuals if a function is calling itself
 1878             // recursively.
 1879             for (j = 0; j < aParamCount; ++j) // For each actual parameter.
 1880             {
 1881                 ExprTokenType &this_param_token = *aParam[j]; // stack[stack_count] is the first actual parameter. A check higher above has already ensured that this line won't cause stack overflow.
 1882                 if (this_param_token.symbol == SYM_VAR && !(j < mParamCount && mParam[j].is_byref))
 1883                 {
 1884                     // Since this formal parameter is passed by value, if it's SYM_VAR, convert it to
 1885                     // a non-var to allow the variables to be backed up and reset further below without
 1886                     // corrupting any SYM_VARs that happen to be locals or params of this very same
 1887                     // function.
 1888                     // DllCall() relies on the fact that this transformation is only done for user
 1889                     // functions, not built-in ones such as DllCall().  This is because DllCall()
 1890                     // sometimes needs the variable of a parameter for use as an output parameter.
 1891                     this_param_token.var->ToToken(this_param_token);
 1892                 }
 1893             }
 1894             // BackupFunctionVars() will also clear each local variable and formal parameter so that
 1895             // if that parameter or local var is assigned a value by any other means during our call
 1896             // to it, new memory will be allocated to hold that value rather than overwriting the
 1897             // underlying recursed/interrupted instance's memory, which it will need intact when it's resumed.
 1898             if (!Var::BackupFunctionVars(*this, aFuncCall.backup, aFuncCall.backup_count)) // Out of memory.
 1899             {
 1900                 aResult = g_script.ScriptError(ERR_OUTOFMEM, mName);
 1901                 return false;
 1902             }
 1903         } // if (func.mInstances > 0)
 1904         //else backup is not needed because there are no other instances of this function on the call-stack.
 1905         // So by definition, this function is not calling itself directly or indirectly, therefore there's no
 1906         // need to do the conversion of SYM_VAR because those SYM_VARs can't be ones that were blanked out
 1907         // due to a function exiting.  In other words, it seems impossible for a there to be no other
 1908         // instances of this function on the call-stack and yet SYM_VAR to be one of this function's own
 1909         // locals or formal params because it would have no legitimate origin.
 1910 
 1911         // Set after above succeeds to ensure local vars are freed and the backup is restored (at some point):
 1912         aFuncCall.func = this;
 1913         
 1914         for (j = 0; j < mParamCount; ++j) // For each formal parameter.
 1915         {
 1916             FuncParam &this_formal_param = mParam[j]; // For performance and convenience.
 1917 
 1918             if (j >= aParamCount || aParam[j]->symbol == SYM_MISSING)
 1919             {
 1920                 if (this_formal_param.is_byref) // v1.0.46.13: Allow ByRef parameters to be optional by converting an omitted-actual into a non-alias formal/local.
 1921                     this_formal_param.var->ConvertToNonAliasIfNecessary(); // Convert from alias-to-normal, if necessary.
 1922 
 1923                 if (param_obj)
 1924                 {
 1925                     ExprTokenType named_value;
 1926                     if (param_obj->GetItem(named_value, this_formal_param.var->mName))
 1927                     {
 1928                         this_formal_param.var->Assign(named_value);
 1929                         continue;
 1930                     }
 1931                 }
 1932             
 1933                 switch(this_formal_param.default_type)
 1934                 {
 1935                 case PARAM_DEFAULT_STR:   this_formal_param.var->Assign(this_formal_param.default_str);    break;
 1936                 case PARAM_DEFAULT_INT:   this_formal_param.var->Assign(this_formal_param.default_int64);  break;
 1937                 case PARAM_DEFAULT_FLOAT: this_formal_param.var->Assign(this_formal_param.default_double); break;
 1938                 default: //case PARAM_DEFAULT_NONE:
 1939                     // No value has been supplied for this REQUIRED parameter.
 1940                     return false; // Abort expression.
 1941                 }
 1942                 continue;
 1943             }
 1944 
 1945             ExprTokenType &token = *aParam[j];
 1946             
 1947             if (!IS_OPERAND(token.symbol)) // Haven't found a way to produce this situation yet, but safe to assume it's possible.
 1948                 return false; // Abort expression.
 1949 
 1950             if (this_formal_param.is_byref)
 1951             {
 1952                 // Note that the previous loop might not have checked things like the following because that
 1953                 // loop never ran unless a backup was needed:
 1954                 if (token.symbol != SYM_VAR)
 1955                 {
 1956                     // L60: Seems more useful and in the spirit of AutoHotkey to allow ByRef parameters
 1957                     // to act like regular parameters when no var was specified.  If we force script
 1958                     // authors to pass a variable, they may pass a temporary variable which is then
 1959                     // discarded, adding a little overhead and impacting the readability of the script.
 1960                     this_formal_param.var->ConvertToNonAliasIfNecessary();
 1961                 }
 1962                 else
 1963                 {
 1964                     this_formal_param.var->UpdateAlias(token.var); // Make the formal parameter point directly to the actual parameter's contents.
 1965                     continue;
 1966                 }
 1967             }
 1968             //else // This parameter is passed "by value".
 1969             // Assign actual parameter's value to the formal parameter (which is itself a
 1970             // local variable in the function).  
 1971             // token.var's Type() is always VAR_NORMAL (e.g. never the clipboard).
 1972             // A SYM_VAR token can still happen because the previous loop's conversion of all
 1973             // by-value SYM_VAR operands into SYM_OPERAND would not have happened if no
 1974             // backup was needed for this function (which is usually the case).
 1975             if (!this_formal_param.var->Assign(token))
 1976             {
 1977                 aResult = FAIL; // Abort thread.
 1978                 return false;
 1979             }
 1980         } // for each formal parameter.
 1981         
 1982         if (mIsVariadic) // i.e. this function is capable of accepting excess params via an object/array.
 1983         {
 1984             // If the caller supplied an array of parameters, copy any key-value pairs with non-numbered keys;
 1985             // otherwise, just create a new object.  Either way, numbered params will be inserted below.
 1986             Object *vararg_obj = param_obj ? param_obj->Clone(true) : Object::Create();
 1987             if (!vararg_obj)
 1988             {
 1989                 aResult = g_script.ScriptError(ERR_OUTOFMEM, mName);
 1990                 return false; // Abort thread.
 1991             }
 1992             if (j < aParamCount)
 1993                 // Insert the excess parameters from the actual parameter list.
 1994                 vararg_obj->InsertAt(0, 1, aParam + j, aParamCount - j);
 1995             // Assign to the "param*" var:
 1996             mParam[mParamCount].var->AssignSkipAddRef(vararg_obj);
 1997         }
 1998 
 1999         DEBUGGER_STACK_PUSH(&aFuncCall)
 2000 
 2001         aResult = Call(&aResultToken); // Call the UDF.
 2002 
 2003         DEBUGGER_STACK_POP()
 2004         
 2005     }
 2006     return (aResult != EARLY_EXIT && aResult != FAIL);
 2007 }
 2008 
 2009 // This is used for maintainability: to ensure it's never forgotten and to reduce code repetition.
 2010 UDFCallInfo::~UDFCallInfo()
 2011 {
 2012     if (func) // func != NULL implies it is a UDF and Var::BackupFunctionVars() has succeeded.
 2013     {
 2014         // Free the memory of all the just-completed function's local variables.  This is done in
 2015         // both of the following cases:
 2016         // 1) There are other instances of this function beneath us on the call-stack: Must free
 2017         //    the memory to prevent a memory leak for any variable that existed prior to the call
 2018         //    we just did.  Although any local variables newly created as a result of our call
 2019         //    technically don't need to be freed, they are freed for simplicity of code and also
 2020         //    because not doing so might result in side-effects for instances of this function that
 2021         //    lie beneath ours that would expect such nonexistent variables to have blank contents
 2022         //    when *they* create it.
 2023         // 2) No other instances of this function exist on the call stack: The memory is freed and
 2024         //    the contents made blank for these reasons:
 2025         //    a) Prevents locals from all being static in duration, and users coming to rely on that,
 2026         //       since in the future local variables might be implemented using a non-persistent method
 2027         //       such as hashing (rather than maintaining a permanent list of Var*'s for each function).
 2028         //    b) To conserve memory between calls (in case the function's locals use a lot of memory).
 2029         //    c) To yield results consistent with when the same function is called while other instances
 2030         //       of itself exist on the call stack.  In other words, it would be inconsistent to make
 2031         //       all variables blank for case #1 above but not do it here in case #2.
 2032         Var::FreeAndRestoreFunctionVars(*func, backup, backup_count);
 2033     }
 2034 }
 2035 
 2036 
 2037 
 2038 ResultType Line::ExpandArgs(ExprTokenType *aResultToken, VarSizeType aSpaceNeeded, Var *aArgVar[])
 2039 // Caller should either provide both or omit both of the parameters.  If provided, it means
 2040 // caller already called GetExpandedArgSize for us.
 2041 // Returns OK, FAIL, or EARLY_EXIT.  EARLY_EXIT occurs when a function-call inside an expression
 2042 // used the EXIT command to terminate the thread.
 2043 {
 2044     // The counterparts of sArgDeref and sArgVar kept on our stack to protect them from recursion caused by
 2045     // the calling of functions in the script:
 2046     LPTSTR arg_deref[MAX_ARGS];
 2047     Var *arg_var[MAX_ARGS];
 2048     int i;
 2049 
 2050     // Make two passes through this line's arg list.  This is done because the performance of
 2051     // realloc() is worse than doing a free() and malloc() because the former often does a memcpy()
 2052     // in addition to the latter's steps.  In addition, realloc() as much as doubles the memory
 2053     // load on the system during the brief time that both the old and the new blocks of memory exist.
 2054     // First pass: determine how much space will be needed to do all the args and allocate
 2055     // more memory if needed.  Second pass: dereference the args into the buffer.
 2056 
 2057     // First pass. It takes into account the same things as 2nd pass.
 2058     size_t space_needed;
 2059     if (aSpaceNeeded == VARSIZE_ERROR)
 2060     {
 2061         space_needed = GetExpandedArgSize(arg_var);
 2062         if (space_needed == VARSIZE_ERROR)
 2063             return FAIL;  // It will have already displayed the error.
 2064     }
 2065     else // Caller already determined it.
 2066     {
 2067         space_needed = aSpaceNeeded;
 2068         for (i = 0; i < mArgc; ++i) // Copying only the actual/used elements is probably faster than using memcpy to copy both entire arrays.
 2069             arg_var[i] = aArgVar[i]; // Init to values determined by caller, which helps performance if any of the args are dynamic variables.
 2070     }
 2071 
 2072     if (space_needed > g_MaxVarCapacity)
 2073         // Dereferencing the variables in this line's parameters would exceed the allowed size of the temp buffer:
 2074         return LineError(ERR_MEM_LIMIT_REACHED);
 2075 
 2076     // Only allocate the buf at the last possible moment,
 2077     // when it's sure the buffer will be used (improves performance when only a short
 2078     // script with no derefs is being run):
 2079     if (space_needed > sDerefBufSize)
 2080     {
 2081         // KNOWN LIMITATION: The memory utilization of *recursive* user-defined functions is rather high because
 2082         // of the size of DEREF_BUF_EXPAND_INCREMENT, which is used to create a new deref buffer for each
 2083         // layer of recursion.  So if a UDF recurses deeply, say 100 layers, about 1600 MB (16KB*100) of
 2084         // memory would be temporarily allocated, which in a worst-case scenario would cause swapping and
 2085         // kill performance.  Perhaps the best solution to this is to dynamically change the size of
 2086         // DEREF_BUF_EXPAND_INCREMENT (via a new global variable) in the expression evaluation section that
 2087         // detects that a UDF has another instance of itself on the call stack.  To ensure proper collapse-back
 2088         // out of nested udfs and threads, the old value should be backed up, the new smaller increment set,
 2089         // then the old size should be passed to FreeAndRestoreFunctionVars() so that it can restore it.
 2090         // However, given the rarity of deep recursion, this doesn't seem worth the extra code size and loss of
 2091         // performance.
 2092         size_t increments_needed = space_needed / DEREF_BUF_EXPAND_INCREMENT;
 2093         if (space_needed % DEREF_BUF_EXPAND_INCREMENT)  // Need one more if above division truncated it.
 2094             ++increments_needed;
 2095         size_t new_buf_size = increments_needed * DEREF_BUF_EXPAND_INCREMENT;
 2096         if (sDerefBuf)
 2097         {
 2098             // Do a free() and malloc(), which should be far more efficient than realloc(), especially if
 2099             // there is a large amount of memory involved here (realloc's ability to do an in-place resize
 2100             // might be unlikely for anything other than small blocks; see compiler's realloc.c):
 2101             free(sDerefBuf);
 2102             if (sDerefBufSize > LARGE_DEREF_BUF_SIZE)
 2103                 --sLargeDerefBufs;
 2104         }
 2105         if (   !(sDerefBuf = tmalloc(new_buf_size))   )
 2106         {
 2107             // Error msg was formerly: "Ran out of memory while attempting to dereference this line's parameters."
 2108             sDerefBufSize = 0;  // Reset so that it can make another attempt, possibly smaller, next time.
 2109             return LineError(ERR_OUTOFMEM); // Short msg since so rare.
 2110         }
 2111         sDerefBufSize = new_buf_size;
 2112         if (sDerefBufSize > LARGE_DEREF_BUF_SIZE)
 2113             ++sLargeDerefBufs;
 2114     }
 2115 
 2116     // Always init our_buf_marker even if zero iterations, because we want to enforce
 2117     // the fact that its prior contents become invalid once we're called.
 2118     // It's also necessary due to the fact that all the old memory is discarded by
 2119     // the above if more space was needed to accommodate this line.
 2120     LPTSTR our_buf_marker = sDerefBuf;  // Prior contents of buffer will be overwritten in any case.
 2121 
 2122     // From this point forward, must not refer to sDerefBuf as our buffer since it might have been
 2123     // given a new memory area by an expression's function-call within this line.  In other words,
 2124     // our_buf_marker is our recursion layer's buffer, but not necessarily sDerefBuf.  To enforce
 2125     // that, and keep responsibility here rather than in ExpandExpression(), set sDerefBuf to NULL
 2126     // so that the zero or more calls to ExpandExpression() made in the loop below (each of which will
 2127     // in turn call zero or more user-defined functions) will allocate and use a single new deref
 2128     // buffer if any of them need it (they all share a single deref buffer because each UDF-call
 2129     // in a particular expression of the current line creates a buf only if necessary, and it won't
 2130     // be necessary if some prior UDF of this same expression or line already created a deref buffer
 2131     // "above" ours because our layer here is the only one who ever frees that upper/extra buffer).
 2132     // Note that it is not possible for a new quasi-thread to directly interrupt ExpandArgs() because
 2133     // ExpandArgs() never calls MsgSleep().  Therefore, each ExpandArgs() layer on the call-stack
 2134     // is safe from interrupting threads overwriting its deref buffer.  It's true that a call to a
 2135     // script function will usually result in MsgSleep(), and thus allow interruptions, but those
 2136     // interruptions would hit some other deref buffer, not that of our layer.
 2137     PRIVATIZE_S_DEREF_BUF;
 2138 
 2139     ResultType result, result_to_return = OK;  // Set default return value.
 2140     Var *the_only_var_of_this_arg;
 2141 
 2142     if (!mArgc)            // v1.0.45: Required by some commands that can have zero parameters (such as Random and
 2143         sArgVar[0] = NULL; // PixelSearch), even if it's just to allow their output-var(s) to be omitted.  This allows OUTPUT_VAR to be used without any need to check mArgC.
 2144     else
 2145     {
 2146         size_t extra_size = our_deref_buf_size - space_needed;
 2147         for (i = 0; i < mArgc; ++i) // For each arg:
 2148         {
 2149             ArgStruct &this_arg = mArg[i]; // For performance and convenience.
 2150 
 2151             // Load-time routines have already ensured that an arg can be an expression only if
 2152             // it's not an input or output var.
 2153             if (this_arg.is_expression)
 2154             {
 2155                 // v1.0.45:
 2156                 // Make ARGVAR1 (OUTPUT_VAR) temporarily valid (the entire array is made valid only later, near the
 2157                 // bottom of this function).  This helps the performance of ACT_ASSIGNEXPR by avoiding the need
 2158                 // resolve a dynamic output variable like "Array%i% := (Expr)" twice: once in GetExpandedArgSize
 2159                 // and again in ExpandExpression()).
 2160                 *sArgVar = *arg_var; // Shouldn't need to be backed up or restored because no one beneath us on the call stack should be using it; only things that go on top of us might overwrite it, so ExpandExpr() must be sure to copy this out before it launches any script-functions.
 2161                 // In addition to producing its return value, ExpandExpression() will alter our_buf_marker
 2162                 // to point to the place in our_deref_buf where the next arg should be written.
 2163                 // In addition, in some cases it will alter some of the other parameters that are arrays or
 2164                 // that are passed by-ref.  Finally, it might temporarily use parts of the buffer beyond
 2165                 // extra_size plus what the size estimator provided for it, so we should be sure here that
 2166                 // everything in our_deref_buf to the right of our_buf_marker is available to it as temporary memory.
 2167                 // Note: It doesn't seem worthwhile to enhance ExpandExpression to give us back a variable
 2168                 // for use in arg_var[] (for performance) because only rarely does an expression yield
 2169                 // a variable other than some function's local variable (and a local's contents are no
 2170                 // longer valid due to having been freed after the call [unless it's static]).
 2171                 arg_deref[i] = ExpandExpression(i, result, mActionType == ACT_RETURN ? aResultToken : NULL  // L31: aResultToken is used to return a non-string value. Pass NULL if mMctionType != ACT_RETURN for maintainability; non-NULL aResultToken should mean we want a token returned - this can be used in future for numeric params or array support in commands.
 2172                     , our_buf_marker, our_deref_buf, our_deref_buf_size, arg_deref, extra_size);
 2173                 extra_size = 0; // See comment below.
 2174                 // v1.0.46.01: The whole point of passing extra_size is to allow an expression to write
 2175                 // a large string to the deref buffer without having to expand it (i.e. if there happens to
 2176                 // be extra room in it that won't be used by ANY arg, including ones after THIS expression).
 2177                 // Since the expression just called above might have used some/all of the extra size,
 2178                 // the line above prevents subsequent expressions in this line from getting any extra size.
 2179                 // It's pretty rare to have more than one expression in a line anyway, and even when there
 2180                 // is there's hardly ever a need for the extra_size.  As an alternative to setting it to
 2181                 // zero, above could check how much the expression wrote to the buffer (by comparing our_buf_marker
 2182                 // before and after the call above), and compare that to how much space was reserved for this
 2183                 // particular arg/expression (which is currently a standard formula for expressions).
 2184                 if (!arg_deref[i])
 2185                 {
 2186                     // A script-function-call inside the expression returned EARLY_EXIT or FAIL.  Report "result"
 2187                     // to our caller (otherwise, the contents of "result" should be ignored since they're undefined).
 2188                     result_to_return = result;
 2189                     goto end;
 2190                 }
 2191                 continue;
 2192             }
 2193 
 2194             if (this_arg.type == ARG_TYPE_OUTPUT_VAR)  // Don't bother wasting the mem to deref output var.
 2195             {
 2196                 // In case its "dereferenced" contents are ever directly examined, set it to be
 2197                 // the empty string.  This also allows the ARG to be passed a dummy param, which
 2198                 // makes things more convenient and maintainable in other places:
 2199                 arg_deref[i] = _T("");
 2200                 continue;
 2201             }
 2202 
 2203             // arg_var[i] was previously set by GetExpandedArgSize() so that we don't have to determine its
 2204             // value again:
 2205             if (   !(the_only_var_of_this_arg = arg_var[i])   ) // Arg isn't an input var or singled isolated deref.
 2206             {
 2207                 #define NO_DEREF (!ArgHasDeref(i + 1))
 2208                 if (NO_DEREF)
 2209                 {
 2210                     arg_deref[i] = this_arg.text;  // Point the dereferenced arg to the arg text itself.
 2211                     continue;  // Don't need to use the deref buffer in this case.
 2212                 }
 2213                 // Otherwise there's more than one variable in the arg, so it must be expanded in the normal,
 2214                 // lower-performance way.
 2215                 arg_deref[i] = our_buf_marker; // Point it to its location in the buffer.
 2216                 if (   !(our_buf_marker = ExpandArg(our_buf_marker, i))   ) // Expand the arg into that location.
 2217                 {
 2218                     result_to_return = FAIL; // ExpandArg() will have already displayed the error.
 2219                     goto end;
 2220                 }
 2221                 continue;
 2222             }
 2223 
 2224             // Since above didn't "continue", the_only_var_of_this_arg==true, so this arg resolves to
 2225             // only a single, naked var.
 2226             switch(ArgMustBeDereferenced(the_only_var_of_this_arg, i, arg_var)) // Yes, it was called by GetExpandedArgSize() too, but a review shows it's difficult to avoid this without being worse than the disease (10/22/2006).
 2227             {
 2228             case CONDITION_FALSE:
 2229                 // This arg contains only a single dereference variable, and no
 2230                 // other text at all.  So rather than copy the contents into the
 2231                 // temp buffer, it's much better for performance (especially for
 2232                 // potentially huge variables like %clipboard%) to simply set
 2233                 // the pointer to be the variable itself.  However, this can only
 2234                 // be done if the var is the clipboard or a non-environment
 2235                 // normal var (since zero-length normal vars need to be fetched via
 2236                 // GetEnvironmentVariable() when g_NoEnv==FALSE).
 2237                 // Update: Changed it so that it will deref the clipboard if it contains only
 2238                 // files and no text, so that the files will be transcribed into the deref buffer.
 2239                 // This is because the clipboard object needs a memory area into which to write
 2240                 // the filespecs it translated:
 2241                 // Update #2: When possible, avoid calling Contents() because that flushes the
 2242                 // cached binary number, which some commands don't need to happen. Only the args that
 2243                 // are specifically written to be optimized should skip it.  Otherwise there would be
 2244                 // problems in things like: date += 31, %Var% (where Var contains "Days")
 2245                 // Update #3: If an expression in an arg after this one causes the var's contents
 2246                 // to be reallocated, it would invalidate any pointer we could get from Contents()
 2247                 // in this iteration.  So instead of calling Contents() here, store a NULL value
 2248                 // as a special indicator for the loop below to call Contents().
 2249                 arg_deref[i] = // The following is ordered for short-circuit performance:
 2250                     (   ACT_IS_ASSIGN(mActionType) && i == 1  // By contrast, for the below i==anything (all args):
 2251                     || (mActionType <= ACT_LAST_OPTIMIZED_IF && mActionType >= ACT_FIRST_OPTIMIZED_IF) // Ordered for short-circuit performance.
 2252                     //|| mActionType == ACT_WHILE // Not necessary to check this one because loadtime leaves ACT_WHILE as an expression in all common cases. Also, there's no easy way to get ACT_WHILE into the range above due to the overlap of other ranges in enum_act.
 2253                     ) && the_only_var_of_this_arg->Type() == VAR_NORMAL // Otherwise, users of this optimization would have to reproduce more of the logic in ArgMustBeDereferenced().
 2254                     ? _T("") : NULL; // See "Update #2" and later comments above.
 2255                 break;
 2256             case CONDITION_TRUE:
 2257                 // the_only_var_of_this_arg is either a reserved var or a normal var of that is also
 2258                 // an environment var (for which GetEnvironmentVariable() is called for), or is used
 2259                 // again in this line as an output variable.  In all these cases, it must
 2260                 // be expanded into the buffer rather than accessed directly:
 2261                 arg_deref[i] = our_buf_marker; // Point it to its location in the buffer.
 2262                 our_buf_marker += the_only_var_of_this_arg->Get(our_buf_marker) + 1; // +1 for terminator.
 2263                 break;
 2264             default: // FAIL should be the only other possibility.
 2265                 result_to_return = FAIL; // ArgMustBeDereferenced() will already have displayed the error.
 2266                 goto end;
 2267             }
 2268         } // for each arg.
 2269 
 2270         // See "Update #3" comment above.  This must be done separately to the loop below since Contents()
 2271         // may cause a warning dialog, which in turn may cause a new thread to launch, thus potentially
 2272         // corrupting sArgDeref/sArgVar.
 2273         for (i = 0; i < mArgc; ++i)
 2274             if (arg_deref[i] == NULL)
 2275                 arg_deref[i] = arg_var[i]->Contents();
 2276 
 2277         // IT'S NOT SAFE to do the following until the above loops FULLY complete because any calls made above to
 2278         // ExpandExpression() might call functions, which in turn might result in a recursive call to ExpandArgs(),
 2279         // which in turn might change the values in the static arrays sArgDeref and sArgVar.
 2280         // Also, only when the loop ends normally is the following needed, since otherwise it's a failure condition.
 2281         // Now that any recursive calls to ExpandArgs() above us on the stack have collapsed back to us, it's
 2282         // safe to set the args of this command for use by our caller, to whom we're about to return.
 2283         for (i = 0; i < mArgc; ++i) // Copying actual/used elements is probably faster than using memcpy to copy both entire arrays.
 2284         {
 2285             sArgDeref[i] = arg_deref[i];
 2286             sArgVar[i] = arg_var[i];
 2287         }
 2288     } // mArgc > 0
 2289 
 2290     // v1.0.40.02: The following loop was added to avoid the need for the ARGn macros to provide an empty
 2291     // string when mArgc was too small (indicating that the parameter is absent).  This saves quite a bit
 2292     // of code size.  Also, the slight performance loss caused by it is partially made up for by the fact
 2293     // that all the other sections don't need to check mArgc anymore.
 2294     // Benchmarks show that it doesn't help performance to try to tweak this with a pre-check such as
 2295     // "if (mArgc < max_params)":
 2296     int max_params = g_act[mActionType].MaxParams; // Resolve once for performance.
 2297     for (i = mArgc; i < max_params; ++i) // START AT mArgc.  For performance, this only does the actual max args for THIS command, not MAX_ARGS.
 2298         sArgDeref[i] = _T("");
 2299         // But sArgVar isn't done (since it's more rarely used) except sArgVar[0] = NULL higher above.
 2300         // Therefore, users of sArgVar must check mArgC if they have any doubt how many args are present in
 2301         // the script line (this is now enforced via macros).
 2302 
 2303     // When the main/large loop above ends normally, it falls into the label below and uses the original/default
 2304     // value of "result_to_return".
 2305 
 2306 end:
 2307     // As of v1.0.31, there can be multiple deref buffers simultaneously if one or more called functions
 2308     // requires a deref buffer of its own (separate from ours).  In addition, if a called function is
 2309     // interrupted by a new thread before it finishes, the interrupting thread will also use the
 2310     // new/separate deref buffer.  To minimize the amount of memory used in such cases,
 2311     // each line containing one or more expression with one or more function call (rather than each
 2312     // function call) will get up to one deref buffer of its own (i.e. only if its function body contains
 2313     // commands that actually require a second deref buffer).  This is achieved by saving sDerefBuf's
 2314     // pointer and setting sDerefBuf to NULL, which effectively makes the original deref buffer private
 2315     // until the line that contains the function-calling expressions finishes completely.
 2316     // Description of recursion and usage of multiple deref buffers:
 2317     // 1) ExpandArgs() receives a line with one or more expressions containing one or more calls to user functions.
 2318     // 2) Worst-case: those function-calls create a new sDerefBuf automatically via us having set sDerefBuf to NULL.
 2319     // 3) Even worse, the bodies of those functions call other functions, which ExpandArgs() receives, resulting in
 2320     //    a recursive leap back to step #1.
 2321     // So the above shows how any number of new deref buffers can be created.  But that's okay as long as the
 2322     // recursion collapses in an orderly manner (or the program exits, in which case the OS frees all its memory
 2323     // automatically).  This is because prior to returning, each recursion layer properly frees any extra deref
 2324     // buffer it was responsible for creating.  It only has to free at most one such buffer because each layer of
 2325     // ExpandArgs() on the call-stack can never be blamed for creating more than one extra buffer.
 2326     // Must always restore the original buffer (if there was one), not keep the new one, because our
 2327     // caller needs the arg_deref addresses, which point into the original buffer.
 2328     DEPRIVATIZE_S_DEREF_BUF;
 2329 
 2330     // For v1.0.31, this is no done right before returning so that any script function calls
 2331     // made by our calls to ExpandExpression() will now be done.  There might still be layers
 2332     // of ExpandArgs() beneath us on the call-stack, which is okay since they will keep the
 2333     // largest of the two available deref bufs (as described above) and thus they should
 2334     // reset the timer below right before they collapse/return.  
 2335     // (Re)set the timer unconditionally so that it starts counting again from time zero.
 2336     // In other words, we only want the timer to fire when the large deref buffer has been
 2337     // unused/idle for a straight 10 seconds.  There is no danger of this timer freeing
 2338     // the deref buffer at a critical moment because:
 2339     // 1) The timer is reset with each call to ExpandArgs (this function);
 2340     // 2) If our ExpandArgs() recursion layer takes a long time to finish, messages
 2341     //    won't be checked and thus the timer can't fire because it relies on the msg loop.
 2342     // 3) If our ExpandArgs() recursion layer launches function-calls in ExpandExpression(),
 2343     //    those calls will call ExpandArgs() recursively and reset the timer if its
 2344     //    buffer (not necessarily the original buffer somewhere on the call-stack) is large
 2345     //    enough.  In light of this, there is a chance that the timer might execute and free
 2346     //    a deref buffer other than the one it was originally intended for.  But in real world
 2347     //    scenarios, that seems rare.  In addition, the consequences seem to be limited to
 2348     //    some slight memory inefficiency.
 2349     // It could be argued that the timer should only be activated when a hypothetical static
 2350     // var sLayers that we maintain here indicates that we're the only layer.  However, if that
 2351     // were done and the launch of a script function creates (directly or through thread
 2352     // interruption, indirectly) a large deref buffer, and that thread is waiting for something
 2353     // such as WinWait, that large deref buffer would never get freed.
 2354     if (sDerefBufSize > LARGE_DEREF_BUF_SIZE)
 2355         SET_DEREF_TIMER(10000) // Reset the timer right before the deref buf is possibly about to become idle.
 2356 
 2357     return result_to_return;
 2358 }
 2359 
 2360     
 2361 
 2362 VarSizeType Line::GetExpandedArgSize(Var *aArgVar[])
 2363 // Returns the size, or VARSIZE_ERROR if there was a problem.
 2364 // This function can return a size larger than what winds up actually being needed
 2365 // (e.g. caused by ScriptGetCursor()), so our callers should be aware that that can happen.
 2366 {
 2367     int i;
 2368     VarSizeType space_needed;
 2369     Var *the_only_var_of_this_arg;
 2370     ResultType result;
 2371 
 2372     // Note: the below loop is similar to the one in ExpandArgs(), so the two should be maintained together:
 2373     for (i = 0, space_needed = 0; i < mArgc; ++i) // FOR EACH ARG:
 2374     {
 2375         ArgStruct &this_arg = mArg[i]; // For performance and convenience.
 2376 
 2377         // Accumulate the total of how much space we will need.
 2378         if (this_arg.type == ARG_TYPE_OUTPUT_VAR)  // These should never be included in the space calculation.
 2379         {
 2380             if (   !(aArgVar[i] = ResolveVarOfArg(i))   ) // v1.0.45: Resolve output variables too, which eliminates a ton of calls to ResolveVarOfArg() in various other functions.  This helps code size more than performance.
 2381                 return VARSIZE_ERROR;  // The above will have already displayed the error.
 2382             continue;
 2383         }
 2384         // Otherwise, set default aArgVar[] (above took care of setting aArgVar[] for itself).
 2385         aArgVar[i] = NULL;
 2386 
 2387         if (this_arg.is_expression)
 2388         {
 2389             // Now that literal strings/numbers are handled by ExpressionToPostfix(), the length used below
 2390             // is more room than is strictly necessary. But given how little space is typically wasted (and
 2391             // that only while the expression is being evaluated), it doesn't seem worth worrying about it.
 2392             // See other comments at macro definition.
 2393             space_needed += EXPR_BUF_SIZE(this_arg.length);
 2394             continue;
 2395         }
 2396 
 2397         // Otherwise:
 2398         // Always do this check before attempting to traverse the list of dereferences, since
 2399         // such an attempt would be invalid in this case:
 2400         the_only_var_of_this_arg = NULL;
 2401         if (this_arg.type == ARG_TYPE_INPUT_VAR) // Previous stage has ensured that arg can't be an expression if it's an input var.
 2402             if (   !(the_only_var_of_this_arg = ResolveVarOfArg(i, false))   )
 2403                 return VARSIZE_ERROR;  // The above will have already displayed the error.
 2404 
 2405         if (!the_only_var_of_this_arg) // It's not an input var.
 2406         {
 2407             if (NO_DEREF)
 2408                 // Don't increase space_needed, even by 1 for the zero terminator, because
 2409                 // the terminator isn't needed if the arg won't exist in the buffer at all.
 2410                 continue;
 2411             // Now we know it has at least one deref.  If the second deref's marker is NULL,
 2412             // the first is the only deref in this arg.  UPDATE: The following will return
 2413             // false for function calls since they are always followed by a set of parentheses
 2414             // (empty or otherwise), thus they will never be seen as isolated by it:
 2415             #define SINGLE_ISOLATED_DEREF (!this_arg.deref[1].marker\
 2416                 && this_arg.deref[0].length == this_arg.length) // and the arg contains no literal text
 2417             if (SINGLE_ISOLATED_DEREF) // This also ensures the deref isn't a function-call.  10/25/2006: It might be possible to avoid the need for detecting SINGLE_ISOLATED_DEREF by transforming them into INPUT_VARs at loadtime.  I almost finished such a mod but the testing and complications with things like ListLines didn't seem worth the tiny benefit.
 2418                 the_only_var_of_this_arg = this_arg.deref[0].var;
 2419         }
 2420         if (the_only_var_of_this_arg) // i.e. check it again in case the above block changed the value.
 2421         {
 2422             // This is set for our caller so that it doesn't have to call ResolveVarOfArg() again, which
 2423             // would a performance hit if this variable is dynamically built and thus searched for at runtime:
 2424             aArgVar[i] = the_only_var_of_this_arg; // For now, this is done regardless of whether it must be dereferenced.
 2425             if (   !(result = ArgMustBeDereferenced(the_only_var_of_this_arg, i, aArgVar))   )
 2426                 return VARSIZE_ERROR;
 2427             if (result == CONDITION_FALSE)
 2428                 continue;
 2429             //else the size of this arg is always included, so fall through to below.
 2430             //else caller wanted it's size unconditionally included, so continue on to below.
 2431             space_needed += the_only_var_of_this_arg->Get() + 1;  // +1 for the zero terminator.
 2432             // NOTE: Get() (with no params) can retrieve a size larger that what winds up actually
 2433             // being needed, so our callers should be aware that that can happen.
 2434             continue;
 2435         }
 2436 
 2437         // Otherwise: This arg has more than one deref, or a single deref with some literal text around it.
 2438         space_needed += this_arg.length + 1; // +1 for this arg's zero terminator in the buffer.
 2439         if (this_arg.deref) // There's at least one deref.
 2440         {
 2441             for (DerefType *deref = this_arg.deref; deref->marker; ++deref)
 2442             {
 2443                 // Replace the length of the deref's literal text with the length of its variable's contents.
 2444                 // At this point, this_arg.is_expression is known to be false. Since non-expressions can't
 2445                 // contain function-calls, there's no need to check deref->is_function.
 2446                 space_needed -= deref->length;
 2447                 space_needed += deref->var->Get(); // If an environment var, Get() will yield its length.
 2448             }
 2449         }
 2450     } // For each arg.
 2451 
 2452     return space_needed;
 2453 }
 2454 
 2455 
 2456 
 2457 ResultType Line::ArgMustBeDereferenced(Var *aVar, int aArgIndex, Var *aArgVar[]) // 10/22/2006: __forceinline didn't help enough to be worth the added code size of having two instances.
 2458 // Shouldn't be called only for args of type ARG_TYPE_OUTPUT_VAR because they never need to be dereferenced.
 2459 // aArgVar[] is used for performance; it's assumed to contain valid items only up to aArgIndex, not beyond
 2460 // (since normally output vars lie to the left of all input vars, so it doesn't seem worth doing anything
 2461 // more complicated).
 2462 // Returns CONDITION_TRUE, CONDITION_FALSE, or FAIL.
 2463 // There are some other functions like ArgLength() and ACT_ADD that have procedures similar to this one, so
 2464 // maintain them together.
 2465 {
 2466     if (mActionType == ACT_SORT) // See PerformSort() for why it's always dereferenced.
 2467         return CONDITION_TRUE;
 2468     aVar = aVar->ResolveAlias(); // Helps performance, but also necessary to accurately detect a match further below.
 2469     VarTypeType aVar_type = aVar->Type();
 2470     if (aVar_type == VAR_CLIPBOARD)
 2471         // Even if the clipboard is both an input and an output var, it still
 2472         // doesn't need to be dereferenced into the temp buffer because the
 2473         // clipboard has two buffers of its own.  The only exception is when
 2474         // the clipboard has only files on it, in which case those files need
 2475         // to be converted into plain text:
 2476         return CLIPBOARD_CONTAINS_ONLY_FILES ? CONDITION_TRUE : CONDITION_FALSE;
 2477     if (aVar_type != VAR_NORMAL || (!g_NoEnv && !aVar->HasContents()) || aVar == g_ErrorLevel) // v1.0.43.08: Added g_NoEnv.
 2478         // Reserved vars must always be dereferenced due to their volatile nature.
 2479         // When g_NoEnv==FALSE, normal vars of length zero are dereferenced because they might exist
 2480         // as system environment variables, whose contents are also potentially volatile (i.e. they
 2481         // are sometimes changed by outside forces).
 2482         // As of v1.0.25.12, g_ErrorLevel is always dereferenced also so that a command that sets ErrorLevel
 2483         // can itself use ErrorLevel as in this example: StringReplace, EndKey, ErrorLevel, EndKey:
 2484         return CONDITION_TRUE;
 2485 
 2486     // Before doing the below, the checks above must be done to ensure it's VAR_NORMAL.  Otherwise, things like
 2487     // the following won't work: StringReplace, o, A_ScriptFullPath, xxx
 2488     // v1.0.45: The following check improves performance slightly by avoiding the loop further below in cases
 2489     // where it's known that a command either doesn't have an output_var or can tolerate the output_var's
 2490     // contents being at the same address as that of one or more of the input-vars.  For example, the commands
 2491     // StringRight/Left and similar can tolerate the same address because they always produce a string whose
 2492     // length is less-than-or-equal to the input-string, thus Assign() will never need to free/realloc the
 2493     // output-var prior to assigning the input-var's contents to it (whose contents are the same as output-var).
 2494     if (!(g_act[mActionType].MaxParamsAu2WithHighBit & 0x80)) // Commands that have this bit don't need final check
 2495         return CONDITION_FALSE;                               // further below (though they do need the ones above).
 2496 
 2497     // Since the above didn't return, we know that this is a NORMAL input var that isn't an
 2498     // environment variable.  Such input vars only need to be dereferenced if they are also
 2499     // used as an output var by the current script line:
 2500     Var *output_var;
 2501     for (int i = 0; i < mArgc; ++i)
 2502         if (i != aArgIndex && mArg[i].type == ARG_TYPE_OUTPUT_VAR)
 2503         {
 2504             if (   !(output_var = (i < aArgIndex) ? aArgVar[i] : ResolveVarOfArg(i, false))   ) // aArgVar: See top of this function for comments.
 2505                 return FAIL;  // It will have already displayed the error.
 2506             if (output_var->ResolveAlias() == aVar)
 2507                 return CONDITION_TRUE;
 2508         }
 2509     // Otherwise:
 2510     return CONDITION_FALSE;
 2511 }
 2512 
 2513 
 2514 
 2515 LPTSTR Line::ExpandArg(LPTSTR aBuf, int aArgIndex, Var *aArgVar) // 10/2/2006: Doesn't seem worth making it inline due to more complexity than expected.  It would also increase code size without being likely to help performance much.
 2516 // Caller must ensure that aArgVar is the variable of the aArgIndex arg when it's of type ARG_TYPE_INPUT_VAR.
 2517 // Caller must be sure not to call this for an arg that's marked as an expression, since
 2518 // expressions are handled by a different function.  Similarly, it must ensure that none
 2519 // of this arg's deref's are function-calls, i.e. that deref->is_function is always false.
 2520 // Caller must ensure that aBuf is large enough to accommodate the translation
 2521 // of the Arg.  No validation of above params is done, caller must do that.
 2522 // Returns a pointer to the char in aBuf that occurs after the zero terminator
 2523 // (because that's the position where the caller would normally resume writing
 2524 // if there are more args, since the zero terminator must normally be retained
 2525 // between args).
 2526 {
 2527     ArgStruct &this_arg = mArg[aArgIndex]; // For performance and convenience.
 2528 #ifdef _DEBUG
 2529     // This should never be called if the given arg is an output var, so flag that in DEBUG mode:
 2530     if (this_arg.type == ARG_TYPE_OUTPUT_VAR)
 2531     {
 2532         LineError(_T("DEBUG: ExpandArg() was called to expand an arg that contains only an output variable."));
 2533         return NULL;
 2534     }
 2535 #endif
 2536 
 2537     if (aArgVar)
 2538         // +1 so that we return the position after the terminator, as required.
 2539         return aBuf += aArgVar->Get(aBuf) + 1;
 2540 
 2541     LPTSTR this_marker, pText = this_arg.text;  // Start at the beginning of this arg's text.
 2542     if (this_arg.deref) // There's at least one deref.
 2543     {
 2544         for (DerefType *deref = this_arg.deref  // Start off by looking for the first deref.
 2545             ; deref->marker; ++deref)  // A deref with a NULL marker terminates the list.
 2546         {
 2547             // FOR EACH DEREF IN AN ARG (if we're here, there's at least one):
 2548             // Copy the chars that occur prior to deref->marker into the buffer:
 2549             for (this_marker = deref->marker; pText < this_marker; *aBuf++ = *pText++); // memcpy() is typically slower for small copies like this, at least on some hardware.
 2550             // Now copy the contents of the dereferenced var.  For all cases, aBuf has already
 2551             // been verified to be large enough, assuming the value hasn't changed between the
 2552             // time we were called and the time the caller calculated the space needed.
 2553             aBuf += deref->var->Get(aBuf); // Caller has ensured that deref->is_function==false
 2554             // Finally, jump over the dereference text. Note that in the case of an expression, there might not
 2555             // be any percent signs within the text of the dereference, e.g. x + y, not %x% + %y%.
 2556             pText += deref->length;
 2557         }
 2558     }
 2559     // Copy any chars that occur after the final deref into the buffer:
 2560     for (; *pText; *aBuf++ = *pText++); // memcpy() is typically slower for small copies like this, at least on some hardware.
 2561     // Terminate the buffer, even if nothing was written into it:
 2562     *aBuf++ = '\0';
 2563     return aBuf; // Returns the position after the terminator.
 2564 }