TailFile Multiple files and Rolling Filename Pattern not working.

[Sumanth Chinthagunta]

Hi 

I am trying to use TailFile Processor to aggregate logs.

my app produce three log files :

1. app-sumo-audit.log

2. app-sumo-api.log

3. app-sumo-logging.log

and corresponding rolling files

app-sumo-audit.log.2017-03-11.0

app-sumo-audit.log.2017-03-11.1

app-sumo-api.log.2017-03-11.0

app-sumo-api.log.2017-03-11.1

app-sumo-logging.log.2017-03-11.0

app-sumo-logging.log.2017-03-11.1

etc.

My TailFile Processor configuration as snow in the attached picture not working for rolled files.

please guide me if I am doing wrong

 

Property

Value

 

Tailing mode

Multiple files

File(s) to Tail

app-sumo-(audit|api|logging).log

Rolling Filename Pattern

${filename}.log.*

Base directory

/Developer/Applications/hNiFi/app/logs

Initial Start Position

Beginning of File

State Location

Local

Recursive lookup

false

Rolling Strategy

Changing name

Lookup frequency

10 minutes

Maximum age

24 hours

 

logback.xml

<?xml version="1.0" encoding="UTF-8"?>

<configuration>

    <include resource="org/springframework/boot/logging/logback/base.xml" />

    <statusListener 

class="ch.qos.logback.core.status.OnConsoleStatusListener" /> 

    <!-- Common default settings -->

    <!-- You can overwrite any defined above property inside logback-<ENV_NAME>.xml -->

    <property name="LOGS_HOME" value="${LOGS_HOME:-./logs}" />

    <property name="MAX_FILE_SIZE" value="${MAX_FILE_SIZE:-200KB}" />

    <property name="MAX_HISTORY" value="${MAX_HISTORY:-5}" />

    <appender name="LOCAL_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">

        <file>${LOGS_HOME}/app-sumo-logging.log</file>

        <encoder class="net.logstash.logback.encoder.LogstashEncoder">

            <fieldNames>

                <version>[ignore]</version>

            </fieldNames>

        </encoder>

        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">

            <fileNamePattern>${LOGS_HOME}/app-sumo-logging.log.%d{yyyy-MM-dd}.%i</fileNamePattern>

              <maxHistory>${MAX_HISTORY}</maxHistory>

            <timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">

                <maxFileSize>${MAX_FILE_SIZE}</maxFileSize>

            </timeBasedFileNamingAndTriggeringPolicy>

        </rollingPolicy>

    </appender>

    <appender name="AUDIT_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">

        <file>${LOGS_HOME}/app-sumo-audit.log</file>

        <encoder class="net.logstash.logback.encoder.LogstashEncoder">

            <fieldNames>

                <version>[ignore]</version>

            </fieldNames>

        </encoder>

        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">

            <fileNamePattern>${LOGS_HOME}/app-sumo-audit.log.%d{yyyy-MM-dd}.%i</fileNamePattern>

            <maxHistory>${MAX_HISTORY}</maxHistory>

            <timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">

                <maxFileSize>${MAX_FILE_SIZE}</maxFileSize>

            </timeBasedFileNamingAndTriggeringPolicy>

        </rollingPolicy>

    </appender>

    <appender name="API_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">

        <file>${LOGS_HOME}/app-sumo-api.log</file>

        <encoder class="net.logstash.logback.encoder.LogstashEncoder">

            <fieldNames>

                <version>[ignore]</version>

                <levelValue>[ignore]</levelValue>

                <thread>[ignore]</thread>

            </fieldNames>

        </encoder>

        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">

            <fileNamePattern>${LOGS_HOME}/app-sumo-api.log.%d{yyyy-MM-dd}.%i</fileNamePattern>

            <maxHistory>${MAX_HISTORY}</maxHistory>

            <timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">

                <maxFileSize>${MAX_FILE_SIZE}</maxFileSize>

            </timeBasedFileNamingAndTriggeringPolicy>

        </rollingPolicy>

    </appender>

        <logger name="org.test.api" level="INFO" additivity="false">

            <appender-ref ref="CONSOLE" />

            <appender-ref ref="API_FILE" />

        </logger>

    <logger name="org.test.Runner" level="INFO" additivity="false">

        <appender-ref ref="CONSOLE" />

        <appender-ref ref="LOCAL_FILE" />

    </logger>

   

    <logger name="org.test.audit" level="INFO" additivity="false">

        <appender-ref ref="CONSOLE" />

        <appender-ref ref="AUDIT_FILE" />

    </logger>

</configuration>

[Andy LoPresto-2]

Hi Sumanth,

Your table text copy/paste did not come through very well, so could you please take a screenshot of the processor properties dialog as it is configured on your system and re-send?

My initial guess is that the issue stems from having multiple dot characters in the filename (I did a brief glance at the code and the regex splits on “\\.”). Can you try a rolling log scheme like app-sumo-audit_2017-03-11-0.log just to see if the logs are now picked up? If so, please confirm and feel free to file a Jira noting the issue and requesting more lenient log capture. I think it’s technically feasible, but it may not have been caught during testing. 

If someone has more context on this or a historical reason this isn’t supported, please weigh in. 

Andy LoPresto

[hidden email]

[hidden email]

PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Mar 11, 2017, at 1:34 AM, Sumanth Chinthagunta <[hidden email]> wrote:

Hi 

I am trying to use TailFile Processor to aggregate logs.

my app produce three log files :

1. app-sumo-audit.log

2. app-sumo-api.log

3. app-sumo-logging.log

and corresponding rolling files

app-sumo-audit.log.2017-03-11.0

app-sumo-audit.log.2017-03-11.1

app-sumo-api.log.2017-03-11.0

app-sumo-api.log.2017-03-11.1

app-sumo-logging.log.2017-03-11.0

app-sumo-logging.log.2017-03-11.1

etc.

My TailFile Processor configuration as snow in the attached picture not working for rolled files.

please guide me if I am doing wrong

 

Property

Value

 

Tailing mode

Multiple files

File(s) to Tail

app-sumo-(audit|api|logging).log

Rolling Filename Pattern

${filename}.log.*

Base directory

/Developer/Applications/hNiFi/app/logs

Initial Start Position

Beginning of File

State Location

Local

Recursive lookup

false

Rolling Strategy

Changing name

Lookup frequency

10 minutes

Maximum age

24 hours

 

logback.xml

<?xml version="1.0" encoding="UTF-8"?>

<configuration>

    <include resource="org/springframework/boot/logging/logback/base.xml" />

    <statusListener 

class="ch.qos.logback.core.status.OnConsoleStatusListener" /> 

    <!-- Common default settings -->

    <!-- You can overwrite any defined above property inside logback-<ENV_NAME>.xml -->

    <property name="LOGS_HOME" value="${LOGS_HOME:-./logs}" />

    <property name="MAX_FILE_SIZE" value="${MAX_FILE_SIZE:-200KB}" />

    <property name="MAX_HISTORY" value="${MAX_HISTORY:-5}" />

    <appender name="LOCAL_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">

        <file>${LOGS_HOME}/app-sumo-logging.log</file>

        <encoder class="net.logstash.logback.encoder.LogstashEncoder">

            <fieldNames>

                <version>[ignore]</version>

            </fieldNames>

        </encoder>

        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">

            <fileNamePattern>${LOGS_HOME}/app-sumo-logging.log.%d{yyyy-MM-dd}.%i</fileNamePattern>

              <maxHistory>${MAX_HISTORY}</maxHistory>

            <timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">

                <maxFileSize>${MAX_FILE_SIZE}</maxFileSize>

            </timeBasedFileNamingAndTriggeringPolicy>

        </rollingPolicy>

    </appender>

    <appender name="AUDIT_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">

        <file>${LOGS_HOME}/app-sumo-audit.log</file>

        <encoder class="net.logstash.logback.encoder.LogstashEncoder">

            <fieldNames>

                <version>[ignore]</version>

            </fieldNames>

        </encoder>

        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">

            <fileNamePattern>${LOGS_HOME}/app-sumo-audit.log.%d{yyyy-MM-dd}.%i</fileNamePattern>

            <maxHistory>${MAX_HISTORY}</maxHistory>

            <timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">

                <maxFileSize>${MAX_FILE_SIZE}</maxFileSize>

            </timeBasedFileNamingAndTriggeringPolicy>

        </rollingPolicy>

    </appender>

    <appender name="API_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">

        <file>${LOGS_HOME}/app-sumo-api.log</file>

        <encoder class="net.logstash.logback.encoder.LogstashEncoder">

            <fieldNames>

                <version>[ignore]</version>

                <levelValue>[ignore]</levelValue>

                <thread>[ignore]</thread>

            </fieldNames>

        </encoder>

        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">

            <fileNamePattern>${LOGS_HOME}/app-sumo-api.log.%d{yyyy-MM-dd}.%i</fileNamePattern>

            <maxHistory>${MAX_HISTORY}</maxHistory>

            <timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">

                <maxFileSize>${MAX_FILE_SIZE}</maxFileSize>

            </timeBasedFileNamingAndTriggeringPolicy>

        </rollingPolicy>

    </appender>

        <logger name="org.test.api" level="INFO" additivity="false">

            <appender-ref ref="CONSOLE" />

            <appender-ref ref="API_FILE" />

        </logger>

    <logger name="org.test.Runner" level="INFO" additivity="false">

        <appender-ref ref="CONSOLE" />

        <appender-ref ref="LOCAL_FILE" />

    </logger>

   

    <logger name="org.test.audit" level="INFO" additivity="false">

        <appender-ref ref="CONSOLE" />

        <appender-ref ref="AUDIT_FILE" />

    </logger>

</configuration>