Accumulate Data in Memory Assertion

The Accumulate Data in Memory Assertion accumulates data (such as metrics) from multiple requests, across different threads, into a buffer that resides in memory on the current node. After this buffer is full or its contents exceed a preset age, the contents are extracted into a Message context variable and the buffer is cleared.

gateway83

The Accumulate Data in Memory Assertion
accumulates data (such as metrics) from multiple requests, across different threads, into a buffer that resides in memory on the current node. After this buffer is full or its contents exceed a preset age, the contents are extracted into a Message context variable and the buffer is cleared.

Unused buffers (those neither appended to nor extracted) older than the value of the bufferdata.maxIdleBufferAge
cluster property are discarded. Scans are performed every 7 minutes.

Example use of this assertion: batch log or metrics information for an external system. Ideal if contacting the external system once per request is infeasible or too expensive.

Policy Sample

The following policy snippet shows how you can accumulate records in a buffer for each request and then send them to an external system after the buffer reaches its size/age limit:

Set Variable rec to "${foo}|${bar}|${baz}%" 
Accumulate Data in Memory to buffer "MyBuffer" from variable ${rec} of up to 10 megabytes for up to 1 hours 
At Least One Assertion Must Evaluate to True:
    All Assertions Must Evaluate to True 
      Comparison: ${buffer.wasExtracted} is a boolean and is true 
      Route via FTP to send ${buffer.extractedMessage} off to some external system 
      <add additional logic to handle instances of FTP upload failure>
    Continue processing

Cluster Properties

All changes to these cluster properties take effect immediately.

Property	Description
accumdata.maxBufferSize	Maximum buffer size that can be specified in the Accumulate Data in Memory Assertion. Default is equivalent to 476MB. Default: 500000000 (bytes)
accumdata.maxIdleBufferAge	Discard unused buffers after this period of time. Default is equivalent to one day. Default: 86400 (seconds)

Context Variables

Variable	Description
<prefix> .wasExtracted	Returns 'true' if the buffer was extracted prior to append; otherwise, returns 'false'.
<prefix> .wasFull	Returns 'true' if the buffer's current size + the appended data size exceeded the buffer size limit specified in the properties; otherwise, returns 'false'.
<prefix> .extractedMessage	Contains the buffer contents if extraction occurred, otherwise it is null.
<prefix> .newAge.millis	Returns the age of the buffer, in milliseconds. If extraction occurred, returns 0 (zero).
<prefix> .newSize.bytes	Returns the size of the buffer. If extraction occurred, this is always equals to the size of the appended data.

Variables are not set if the assertion fails with the status "SERVER_ERROR".

Properties

Setting	What you should know...
Buffer Name	Buffers are created per node. The buffer will be created if it does not already exist. Creating too many buffers may cause the Gateway to run out of memory. For example, using context variables in the name may generate many buffers: ${requestId} (creates buffers based on the request ID) ${request.authenticateduser} (creates buffers based on authenticated user names
Append Data from Variable	Contains data to be added to the buffer. This variable is created elsewhere in the policy and should exist at the time this assertion is run. Example: Consider a context variable that contains a single line of pipe-delimited data such as the following: 2014\|Dec\|1\|12\|43\|Joe\|Smith\|Canada\|Racing Truck\|Red\r\n (This sample contains a carriage return and line feed.) You can then use the Accumulate Data in Memory assertion to append this line of data to the buffer.
Maximum Buffer Time	Controls how old the oldest buffered data is permitted to be for the assertion to succeed. If the buffer contains data older than this at the time the assertion runs, then the buffer is extracted to <prefix>.extractedMessage prior to the append and then <prefix>.wasExtracted is set to "true".
Maximum Buffer Size	Controls how much memory the buffer may consume on a node. If the current buffer size plus the appended data size exceeds this limit, then the buffer is extracted prior to the append and placed into the <prefix>.extractedMessage variable. If the appended data size alone exceeds this limit, then the assertion fails and the buffer remains in its current state (no appending occurs). The maximum value that can be entered here is determined by the accumdata.maxBufferSize cluster property. WARNING: Exercise caution if using very large buffers. If the back-end server becomes unresponsive while the Gateway is accumulating large amounts of data, and the buffer can't be emptied, this may cause the Gateway to run out of memory.
Variable Prefix	A prefix is required.

Frequently Asked Questions

Question	Answer
Can I modify the existing buffer contents?	No. To preserve thread safety, the only operation permitted is to append data to the buffer.
Will collecting this data in the in-memory buffer affect performance?	Not generally. However very large or numerous buffers may fill the Gateway's memory, which will affect performance. Pay attention to the buffer size and the number of buffers created (based on context variables in the buffer name).
What is the content-type of the Message variable that holds the extracted buffer?	The content-type is "application/octet-stream". You can change this using the Validate or Change Content Type Assertion.
Why do my context variables have no values?	The variables are not set if the assertion fails with the status "SERVER_ERROR".

Additional Information

When using the Accumulate Data in Memory Assertion for log records, special attention is required to ensure line breaks are handled correctly. Since the data is stored in binary format, you must be explicit in adding the line breaks. The simplest way to do this is to create a context variable such as ${crlf}
that contains the binary equivalent to 'crlf' and then append that to the text being accumulated every time. For example:

View above policy as XML code

<?xml version="1.0" encoding="UTF-8"?>
<wsp:Policy xmlns:L7p="http://www.layer7tech.com/ws/policy" xmlns:wsp="http://schemas.xmlsoap.org/ws/2002/12/policy">
    <wsp:All wsp:Usage="Required">
        <L7p:SetVariable>
            <L7p:Base64Expression stringValue="JTBEJTBB"/>
            <L7p:VariableToSet stringValue="crlf"/>
        </L7p:SetVariable>
        <L7p:EncodeDecode>
            <L7p:SourceVariableName stringValue="crlf"/>
            <L7p:TargetDataType variableDataType="string"/>
            <L7p:TargetVariableName stringValue="crlf"/>
            <L7p:TransformType transformType="URL_DECODE"/>
        </L7p:EncodeDecode>
        <L7p:SetVariable>
            <L7p:Base64Expression stringValue="QXJiaXRyYXJ5IFN0cmluZyR7Y3JsZn0="/>
            <L7p:VariableToSet stringValue="record"/>
        </L7p:SetVariable>
        <L7p:BufferData>
            <L7p:BufferName stringValue="testBuffer"/>
            <L7p:MaxSizeBytes longValue="50"/>
            <L7p:NewDataVarName stringValue="record"/>
        </L7p:BufferData>
    </wsp:All>
</wsp:Policy>

An alternative is to use a newline character instead, which removes the need to decode crlf characters. However this is more difficult to troubleshoot and cannot be illustrated correctly due to the invisible trailing crlf character.

View XML code for alternate policy

?<?xml version="1.0" encoding="UTF-8"?>
<wsp:Policy xmlns:L7p="http://www.layer7tech.com/ws/policy" xmlns:wsp="http://schemas.xmlsoap.org/ws/2002/12/policy">
    <wsp:All wsp:Usage="Required">
        <L7p:SetVariable>
            <L7p:Base64Expression stringValue="QXJiaXRyYXJ5IFN0cmluZw0K"/>
            <L7p:VariableToSet stringValue="record"/>
        </L7p:SetVariable>
        <L7p:BufferData>
            <L7p:BufferName stringValue="testBuffer"/>
            <L7p:MaxSizeBytes longValue="50"/>
            <L7p:NewDataVarName stringValue="record"/>
        </L7p:BufferData>
    </wsp:All>
</wsp:Policy>

Dealing with text in the buffer requires converting ${buffer.extractedMessage}
from 'application/octet-stream' to 'text/plain'. As mentioned in the FAQ section above, you use the Validate or Change Content Type Assertion (Threat Protection) (with the "Re-initialize message" option selected) to do this. However, you should include logic to ensure that this context variable exists:

View above policy as XML code

<?xml version="1.0" encoding="UTF-8"?>
<wsp:Policy xmlns:L7p="http://www.layer7tech.com/ws/policy" xmlns:wsp="http://schemas.xmlsoap.org/ws/2002/12/policy">
    <wsp:All wsp:Usage="Required">
        <L7p:BufferData>
            <L7p:BufferName stringValue="testBuffer"/>
            <L7p:MaxSizeBytes longValue="50"/>
            <L7p:NewDataVarName stringValue="record"/>
        </L7p:BufferData>
        <wsp:OneOrMore wsp:Usage="Required">
            <L7p:ComparisonAssertion>
                <L7p:CaseSensitive booleanValue="false"/>
                <L7p:Expression1 stringValue="${buffer.wasExtracted}"/>
                <L7p:Operator operatorNull="null"/>
                <L7p:Predicates predicates="included">
                    <L7p:item dataType="included">
                        <L7p:Type variableDataType="boolean"/>
                    </L7p:item>
                    <L7p:item binary="included">
                        <L7p:CaseSensitive booleanValue="false"/>
                        <L7p:RightValue stringValue="false"/>
                    </L7p:item>
                </L7p:Predicates>
            </L7p:ComparisonAssertion>
            <L7p:ContentType>
                <L7p:ChangeContentType booleanValue="true"/>
                <L7p:NewContentTypeValue stringValue="text/plain"/>
                <L7p:OtherTargetMessageVariable stringValue="buffer.extractedMessage"/>
                <L7p:ReinitializeMessage booleanValue="true"/>
                <L7p:Target target="OTHER"/>
            </L7p:ContentType>
        </wsp:OneOrMore>
    </wsp:All>
</wsp:Policy>

Logging, Auditing, and Alerts Assertions

Content feedback and comments

CA API Gateway 9.1

Accumulate Data in Memory Assertion