TailFile Multiple files and Rolling Filename Pattern not working.

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

TailFile Multiple files and Rolling Filename Pattern not working.

Sumanth Chinthagunta
Hi 
I am trying to use TailFile Processor to aggregate logs.
my app produce three log files :
1. app-sumo-audit.log
2. app-sumo-api.log
3. app-sumo-logging.log

and corresponding rolling files

app-sumo-audit.log.2017-03-11.0
app-sumo-audit.log.2017-03-11.1
app-sumo-api.log.2017-03-11.0
app-sumo-api.log.2017-03-11.1
app-sumo-logging.log.2017-03-11.0
app-sumo-logging.log.2017-03-11.1
etc.

My TailFile Processor configuration as snow in the attached picture not working for rolled files.

please guide me if I am doing wrong
 
Property
Value
 
Tailing mode
Multiple files
File(s) to Tail
app-sumo-(audit|api|logging).log
Rolling Filename Pattern
${filename}.log.*
Base directory
/Developer/Applications/hNiFi/app/logs
Initial Start Position
Beginning of File
State Location
Local
Recursive lookup
false
Rolling Strategy
Changing name
Lookup frequency
10 minutes
Maximum age
24 hours
 



logback.xml

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <include resource="org/springframework/boot/logging/logback/base.xml" />
    <statusListener 
class="ch.qos.logback.core.status.OnConsoleStatusListener" /> 

    <!-- Common default settings -->
    <!-- You can overwrite any defined above property inside logback-<ENV_NAME>.xml -->
    <property name="LOGS_HOME" value="${LOGS_HOME:-./logs}" />
    <property name="MAX_FILE_SIZE" value="${MAX_FILE_SIZE:-200KB}" />
    <property name="MAX_HISTORY" value="${MAX_HISTORY:-5}" />

    <appender name="LOCAL_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>${LOGS_HOME}/app-sumo-logging.log</file>
        <encoder class="net.logstash.logback.encoder.LogstashEncoder">
            <fieldNames>
                <version>[ignore]</version>
            </fieldNames>
        </encoder>

        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <fileNamePattern>${LOGS_HOME}/app-sumo-logging.log.%d{yyyy-MM-dd}.%i</fileNamePattern>
              <maxHistory>${MAX_HISTORY}</maxHistory>
            <timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
                <maxFileSize>${MAX_FILE_SIZE}</maxFileSize>
            </timeBasedFileNamingAndTriggeringPolicy>
        </rollingPolicy>
    </appender>

    <appender name="AUDIT_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>${LOGS_HOME}/app-sumo-audit.log</file>
        <encoder class="net.logstash.logback.encoder.LogstashEncoder">
            <fieldNames>
                <version>[ignore]</version>
            </fieldNames>
        </encoder>

        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <fileNamePattern>${LOGS_HOME}/app-sumo-audit.log.%d{yyyy-MM-dd}.%i</fileNamePattern>
            <maxHistory>${MAX_HISTORY}</maxHistory>
            <timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
                <maxFileSize>${MAX_FILE_SIZE}</maxFileSize>
            </timeBasedFileNamingAndTriggeringPolicy>
        </rollingPolicy>
    </appender>

    <appender name="API_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>${LOGS_HOME}/app-sumo-api.log</file>
        <encoder class="net.logstash.logback.encoder.LogstashEncoder">
            <fieldNames>
                <version>[ignore]</version>
                <levelValue>[ignore]</levelValue>
                <thread>[ignore]</thread>
            </fieldNames>
        </encoder>

        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <fileNamePattern>${LOGS_HOME}/app-sumo-api.log.%d{yyyy-MM-dd}.%i</fileNamePattern>
            <maxHistory>${MAX_HISTORY}</maxHistory>
            <timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
                <maxFileSize>${MAX_FILE_SIZE}</maxFileSize>
            </timeBasedFileNamingAndTriggeringPolicy>
        </rollingPolicy>
    </appender>

        <logger name="org.test.api" level="INFO" additivity="false">
            <appender-ref ref="CONSOLE" />
            <appender-ref ref="API_FILE" />
        </logger>

    <logger name="org.test.Runner" level="INFO" additivity="false">
        <appender-ref ref="CONSOLE" />
        <appender-ref ref="LOCAL_FILE" />
    </logger>
   
    <logger name="org.test.audit" level="INFO" additivity="false">
        <appender-ref ref="CONSOLE" />
        <appender-ref ref="AUDIT_FILE" />
    </logger>

</configuration>
Reply | Threaded
Open this post in threaded view
|

Re: TailFile Multiple files and Rolling Filename Pattern not working.

Andy LoPresto-2
Hi Sumanth,

Your table text copy/paste did not come through very well, so could you please take a screenshot of the processor properties dialog as it is configured on your system and re-send?

My initial guess is that the issue stems from having multiple dot characters in the filename (I did a brief glance at the code and the regex splits on “\\.”). Can you try a rolling log scheme like app-sumo-audit_2017-03-11-0.log just to see if the logs are now picked up? If so, please confirm and feel free to file a Jira noting the issue and requesting more lenient log capture. I think it’s technically feasible, but it may not have been caught during testing. 

If someone has more context on this or a historical reason this isn’t supported, please weigh in. 

Andy LoPresto
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Mar 11, 2017, at 1:34 AM, Sumanth Chinthagunta <[hidden email]> wrote:

Hi 
I am trying to use TailFile Processor to aggregate logs.
my app produce three log files :
1. app-sumo-audit.log
2. app-sumo-api.log
3. app-sumo-logging.log

and corresponding rolling files

app-sumo-audit.log.2017-03-11.0
app-sumo-audit.log.2017-03-11.1
app-sumo-api.log.2017-03-11.0
app-sumo-api.log.2017-03-11.1
app-sumo-logging.log.2017-03-11.0
app-sumo-logging.log.2017-03-11.1
etc.

My TailFile Processor configuration as snow in the attached picture not working for rolled files.

please guide me if I am doing wrong
 
Property
Value
 
Tailing mode
Multiple files
File(s) to Tail
app-sumo-(audit|api|logging).log
Rolling Filename Pattern
${filename}.log.*
Base directory
/Developer/Applications/hNiFi/app/logs
Initial Start Position
Beginning of File
State Location
Local
Recursive lookup
false
Rolling Strategy
Changing name
Lookup frequency
10 minutes
Maximum age
24 hours
 



logback.xml

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <include resource="org/springframework/boot/logging/logback/base.xml" />
    <statusListener 
class="ch.qos.logback.core.status.OnConsoleStatusListener" /> 

    <!-- Common default settings -->
    <!-- You can overwrite any defined above property inside logback-<ENV_NAME>.xml -->
    <property name="LOGS_HOME" value="${LOGS_HOME:-./logs}" />
    <property name="MAX_FILE_SIZE" value="${MAX_FILE_SIZE:-200KB}" />
    <property name="MAX_HISTORY" value="${MAX_HISTORY:-5}" />

    <appender name="LOCAL_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>${LOGS_HOME}/app-sumo-logging.log</file>
        <encoder class="net.logstash.logback.encoder.LogstashEncoder">
            <fieldNames>
                <version>[ignore]</version>
            </fieldNames>
        </encoder>

        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <fileNamePattern>${LOGS_HOME}/app-sumo-logging.log.%d{yyyy-MM-dd}.%i</fileNamePattern>
              <maxHistory>${MAX_HISTORY}</maxHistory>
            <timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
                <maxFileSize>${MAX_FILE_SIZE}</maxFileSize>
            </timeBasedFileNamingAndTriggeringPolicy>
        </rollingPolicy>
    </appender>

    <appender name="AUDIT_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>${LOGS_HOME}/app-sumo-audit.log</file>
        <encoder class="net.logstash.logback.encoder.LogstashEncoder">
            <fieldNames>
                <version>[ignore]</version>
            </fieldNames>
        </encoder>

        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <fileNamePattern>${LOGS_HOME}/app-sumo-audit.log.%d{yyyy-MM-dd}.%i</fileNamePattern>
            <maxHistory>${MAX_HISTORY}</maxHistory>
            <timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
                <maxFileSize>${MAX_FILE_SIZE}</maxFileSize>
            </timeBasedFileNamingAndTriggeringPolicy>
        </rollingPolicy>
    </appender>

    <appender name="API_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>${LOGS_HOME}/app-sumo-api.log</file>
        <encoder class="net.logstash.logback.encoder.LogstashEncoder">
            <fieldNames>
                <version>[ignore]</version>
                <levelValue>[ignore]</levelValue>
                <thread>[ignore]</thread>
            </fieldNames>
        </encoder>

        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <fileNamePattern>${LOGS_HOME}/app-sumo-api.log.%d{yyyy-MM-dd}.%i</fileNamePattern>
            <maxHistory>${MAX_HISTORY}</maxHistory>
            <timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
                <maxFileSize>${MAX_FILE_SIZE}</maxFileSize>
            </timeBasedFileNamingAndTriggeringPolicy>
        </rollingPolicy>
    </appender>

        <logger name="org.test.api" level="INFO" additivity="false">
            <appender-ref ref="CONSOLE" />
            <appender-ref ref="API_FILE" />
        </logger>

    <logger name="org.test.Runner" level="INFO" additivity="false">
        <appender-ref ref="CONSOLE" />
        <appender-ref ref="LOCAL_FILE" />
    </logger>
   
    <logger name="org.test.audit" level="INFO" additivity="false">
        <appender-ref ref="CONSOLE" />
        <appender-ref ref="AUDIT_FILE" />
    </logger>

</configuration>


signature.asc (859 bytes) Download Attachment