apim_4xx_support_multiple_analytics_publishers APIM manage workflow with multiple roles APIM 3.0.0 per API based subscription workflow Logging internal HTTP requests Log APIM analytics events to a file Monetization and sample with WSO2 API Manager 2.6.0 Share application and subscription among a set of specific groups or roles WSO2 APIM Correlating analytics event with correlationID APIM analytics distinguish production and sandbox traffic APIM 2.x.x analytics internal and analytics tuneup Configure APIM(Next release) Key Manager User stores APIM(Next release) working with key manager DAS 3.x Parse system variables to Spark Context Revoke OAuth application In APIM 2.1.0 Next WSO2 APIM powered by WSO2 Ballerina Configure WSO2 APIM Analytics on Cluster environment Configure WSO2 DAS 3.1.0 for WSO2 APIM 2.0.0 Analytics WSO2 APIM publishing custom statistics WSO2 APIM Error codes Working with WSO2 message tracer Use DAS admin service to query using Spark SQL Configure WSO2 APIM Analytics using XML WSO2 APIM Generating and Retrieving Custom Statistics Understanding WSO2 APIM Statistics Model Publishing WSO2 APIM 1.10.x Runtime Statistics to DAS with RDBMS Publishing_APIM_1100_Runtime_Statistics_to_DAS Aggregate functions with WSO2 DAS REST API Create a cApp for WSO2 DAS Debugging WSO2 Products using OSGI console. Publishing APIM Runtime Statistics to DAS Deploy cApp on WSO2 DAS How to configure and start the Accumulo minicluster How to setup DNS server on Ubuntu and Ubuntu server How to use Java Reflection how to install apache web server on ubuntu and ubuntu server How to install Mail server on Ubuntu and Ubuntu server How to install squirrelmail webmail client on Ubuntu and Ubuntu Server Pass and return String value to JNI method Pass and return numeric value to JNI method Calling a C Function from the Java Programming Language using JNI AXIS 2 Sample web service Client with maven and eclipse How to setup AXIS 2 with Apache Tomcat AXIS 2 Sample web service with maven and eclipse Robot framework Sample with Selenium Robot framework Custom Library Sample Behaviour-Driven Development with JBehave and Eclipse Play Audio with Netbeans and linking with LibVLC Implement LibVLC based player with QT-part2 Simple Audio playing sample with LibVLC How to install LibVLC on Ubuntu Implement LibVLC based player with QT-part1
Log APIM analytics events to a file
  1. Introduction

    APIM Gateway tracks the analytics data if analytics is enabled and configured. But in order to use this feature APIM analytics need to be configured. If analytics is correctly configured these data processed and generate summary data that can be visualized in publisher and store statistics apps. But there can be requirements to get these analytics data without configuring apim analytics. Because adding another analytics server or deployment to the existing setup may need additional changes and required additional resources. As a solution, you can get these data to a file similar to the logging.

    For this, we can add a custom event publisher using existing extension points and publish these data to the logger instead of the analyzer. So we can use the APIM component to a maven project and apply the configuration in the gateway node.

  2. Implementation
    • Create a maven project
    • Include required repository and maven dependency
      <?xml version="1.0" encoding="UTF-8"?>
      <project xmlns="http://maven.apache.org/POM/4.0.0"
               xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
               xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
          <modelVersion>4.0.0</modelVersion>
      
          <groupId>com.rukspot.samples</groupId>
          <artifactId>com.rukspot.samples.log_analytics</artifactId>
          <version>1.0-SNAPSHOT</version>
      
          <dependencies>
              <dependency>
                  <groupId>org.wso2.carbon.apimgt</groupId>
                  <artifactId>org.wso2.carbon.apimgt.usage.publisher</artifactId>
                  <version>6.4.50</version>
              </dependency>
          </dependencies>
      
          <repositories>
              <repository>
                  <id>wso2-nexus</id>
                  <name>WSO2 internal Repository</name>
                  <url>http://maven.wso2.org/nexus/content/groups/wso2-public/</url>
                  <releases>
                      <enabled>true</enabled>
                      <updatePolicy>daily</updatePolicy>
                      <checksumPolicy>ignore</checksumPolicy>
                  </releases>
              </repository>
          </repositories>
      </project>
      				
    • Create a class which is extended to org.wso2.carbon.apimgt.usage.publisher.APIMgtUsageDataBridgeDataPublisher
    • Override all the method and implement nothing to ignore its functions
    • Log all the required data by converting it to a JSON
      package com.rukspot.samples.analytics.publisher;
      
      import com.google.gson.Gson;
      import org.apache.commons.logging.Log;
      import org.apache.commons.logging.LogFactory;
      import org.wso2.carbon.apimgt.api.APIManagementException;
      import org.wso2.carbon.apimgt.usage.publisher.APIMgtUsageDataBridgeDataPublisher;
      import org.wso2.carbon.apimgt.usage.publisher.dto.AlertTypeDTO;
      import org.wso2.carbon.apimgt.usage.publisher.dto.FaultPublisherDTO;
      import org.wso2.carbon.apimgt.usage.publisher.dto.RequestResponseStreamDTO;
      import org.wso2.carbon.apimgt.usage.publisher.dto.ThrottlePublisherDTO;
      
      public class FilePublisher extends APIMgtUsageDataBridgeDataPublisher {
          private static final Log log = LogFactory.getLog(FilePublisher.class);
          private Gson gson = new Gson();
      
          public FilePublisher() {
              // Do nothing
          }
      
          @Override
          public void init() {
              // Do nothing
          }
      
          @Override
          public void publishEvent(FaultPublisherDTO faultPublisherDTO) {
              log.info(gson.toJson(faultPublisherDTO));
          }
      
          @Override
          public void publishEvent(ThrottlePublisherDTO throttPublisherDTO) {
              log.info(gson.toJson(throttPublisherDTO));
          }
      
          @Override
          public void publishEvent(AlertTypeDTO alertTypeDTO) throws APIManagementException {
              // Required only to log alert configurations
              // log.info(gson.toJson(alertTypeDTO));
          }
      
          @Override
          public void publishEvent(RequestResponseStreamDTO requestStream) {
              log.info(gson.toJson(requestStream));
          }
      }
      
      				
  3. Configuration
    • Download and extract WSO2 API Manager 2.6.0
    • Enable analytics in wso2 Gateway
    • Disable workflow publishing in order to disable connectivity check to the Analytics
    • Define the custom publisher in <AM_HOME>/repository/conf/api-manager.xml
      <Analytics>
          <!-- Enable Analytics for API Manager -->
          <Enabled>false</Enabled>
      
          <!--Skip workflow data publisher initiation and even publishing-->
          <SkipWorkflowEventPublisher>true</SkipWorkflowEventPublisher>
      
          <!-- API Usage Data Publisher. -->
          <PublisherClass>org.wso2.carbon.apimgt.usage.publisher.APIMgtUsageDataBridgeDataPublisher</PublisherClass>
      
          ......
      </Analytics>
      				
    • Define the package level to be log in <AM_HOME>/repository/conf/log4j.properties
      log4j.category.com.rukspot.samples.analytics.publisher=INFO
      				
    • Build the source code with 'mvn clean install'
    • Copy the build jar or from github to the <AM_HOME>/repository/components/lib
  4. Testing
    • Restart the server once above configurations are applied
    • Deploy an API and get a token once subscribe
    • Invoke the API and observe the log in <AM_HOME>/repository/wso2carbon.log
      [2019-04-06 19:04:20,658]  INFO - FilePublisher {"applicationConsumerKey":"waxwrjrA2vWIqnxDOJ0PST7IXLwa","applicationName":"DefaultApplication","applicationId":"1","applicationOwner":"admin","apiContext":"/pizzashack/1.0.0","apiName":"PizzaShackAPI","apiVersion":"1.0.0","apiResourcePath":"/menu","apiResourceTemplate":"/menu","apiMethod":"GET","apiCreator":"admin","apiCreatorTenantDomain":"carbon.super","apiTier":"Unlimited","apiHostname":"172.17.0.1","username":"admin@carbon.super","userTenantDomain":"carbon.super","userIp":"10.10.10.2","userAgent":"curl/7.47.0","serviceTime":3,"requestTimestamp":1554557657696,"throttledOut":false,"backendTime":20,"responseCacheHit":false,"responseSize":0,"protocol":"https-8243","responseCode":200,"destination":"https://localhost:9443/am/sample/pizzashack/v1/api/","metaClientType":"PRODUCTION","responseTime":23,"gatewayType":"SYNAPSE","correlationID":"5fe11502-8d98-439d-9056-8b5762236f4a","label":"Synapse","executionTime":{"securityLatency":0,"throttlingLatency":0,"requestMediationLatency":2,"responseMediationLatency":0,"backEndLatency":20,"otherLatency":0}}
      
      [2019-04-06 19:07:43,363]  INFO - FilePublisher {"applicationConsumerKey":"waxwrjrA2vWIqnxDOJ0PST7IXLwa","applicationName":"DefaultApplication","applicationId":"1","applicationOwner":"admin","apiContext":"/pizzashack/1.0.0","apiName":"PizzaShackAPI","apiVersion":"1.0.0","apiResourcePath":"/menu","apiResourceTemplate":"/menu","apiMethod":"GET","apiCreator":"admin","apiCreatorTenantDomain":"carbon.super","apiTier":"Unlimited","apiHostname":"172.17.0.1","username":"admin@carbon.super","userTenantDomain":"carbon.super","userIp":"10.10.10.2","userAgent":"curl/7.47.0","serviceTime":4,"requestTimestamp":1554557863334,"throttledOut":false,"backendTime":17,"responseCacheHit":false,"responseSize":0,"protocol":"https-8243","responseCode":200,"destination":"https://localhost:9443/am/sample/pizzashack/v1/api/","metaClientType":"PRODUCTION","responseTime":21,"gatewayType":"SYNAPSE","correlationID":"cb93d815-d738-4879-8305-2124fb48ff39","label":"Synapse","executionTime":{"securityLatency":3,"throttlingLatency":0,"requestMediationLatency":0,"responseMediationLatency":0,"backEndLatency":17,"otherLatency":0}}
      
      				
  5. Get the Project
    • Maven project is available on github

Add Comment

* Required information
1000
Powered by Commentics

Comments (3)

Sort By
Topic: Log APIM analytics events to a file
Gravatar
New
Reza
Gravatar
4
1
0
Nov 2019
First Poster
Reza says...

I just set values below in the file "<AM_HOME>/repository/conf/api-manager.xml"

<Analytics>

<Enabled>false</Enabled>

<SkipWorkflowEventPublisher>true</SkipWorkflowEventPublisher>

</Analytics>

and also just added the line below at the end of the "log4j.properties" file.

But I couldn't get the logs in the log file.

What should I do? My wso2 AM version is 2.6, if I could run it successfully, I will build it for version 3 as well.

Gravatar
New
Rukshan
Gravatar
6
4
0
Dec 2018
Top Poster
Most Likes
Rukshan says...

Can you set the config like this? It should work in 2.6.0.

<Analytics>

<!-- Enable Analytics for API Manager -->

<Enabled>true</Enabled>

<!--Skip workflow data publisher initiation and even publishing-->

<SkipWorkflowEventPublisher>true</SkipWorkflowEventPublisher>

<!-- API Usage Data Publisher. -->

<PublisherClass>com.rukspot.samples.analytics.publisher. FilePublisher</PublisherClass>

......

</Analytics>

Gravatar
New
Randima Somathilaka
Gravatar
1
0
0
Nov 2019
Randima Somathilaka says...

In the Configuration section in the post, Shouldn't the api-manager.xml is looked like below?

<Analytics>

<!-- Enable Analytics for API Manager -->

<Enabled>true</Enabled>

<!--Skip workflow data publisher initiation and even publishing-->

<SkipWorkflowEventPublisher>true</SkipWorkflowEventPublisher>

<!-- API Usage Data Publisher. -->

<PublisherClass>com.rukspot.samples.analytics.publisher. FilePublisher</PublisherClass>

......

</Analytics>

Gravatar
New
Rukshan
Gravatar
6
4
0
Dec 2018
Top Poster
Most Likes
Rukshan says...

yes. you need to set <PublisherClass>com.rukspot.samples.analytics.publisher. FilePublisher</PublisherClass>

Gravatar
New
Reza
Gravatar
4
1
0
Nov 2019
First Poster
Reza says...

Is it work in AM 3?

Gravatar
New
Rukshan
Gravatar
6
4
0
Dec 2018
Top Poster
Most Likes
Rukshan says...

yes it work with AM 3.0.0. But make sure to use carbon.apimgt.version as 6.5.349

Gravatar
New
Reza
Gravatar
4
1
0
Nov 2019
First Poster
Reza says...

In the AM 3.0, every call raise the attached error.

I changed the dependency as you said:

<dependency>

<groupId>org.wso2.carbon.apimgt</groupId>

<artifactId>org.wso2.carbon.apimgt.usage.publisher</artifactId>

<version>6.5.349</version>

</dependency>

Upload
Gravatar
New
Rukshan
Gravatar
6
4
0
Dec 2018
Top Poster
Most Likes
Rukshan says...

This is not an analytics related issue. This is issues happen when publishing throttle related information to Traffic manager. Seems your client IP is an IPv6 address and hence the issue. Is 'X-Forwarded-For' header is present in your request and is it ipv6 address? Can you send an ipv4 address with 'X-Forwarded-For' header in your request?

Gravatar
New
Reza
Gravatar
4
1
0
Nov 2019
First Poster
Reza says...

I just deployed the sample after setting the configurations in the deployment.toml, but the configuration in api-manager.xml won't change at all.

here is my deployment.toml config:

#[apim.analytics]

enable = true

event_publisher_impl = "com.rukspot.samples.analytics.publisher.FilePublisher"

and in the log4j2.properties:

log4j.category.com.rukspot.samples.analytics.publisher=INFO

I am testing the sample API using the swagger in the "Try Out" section and got that error you saw, the service works correctly but I don't receive any log in my console. If I use the curl and invoke the service via curl, it does not generate any error in the logs, but still I don't get any analytics log either.

Gravatar
New
Rukshan
Gravatar
6
4
0
Dec 2018
Top Poster
Most Likes
Rukshan says...

github project is updated for the APIM 300. please find and follow the instruction given in [1]

[1] https://github.com/ruks/logs-analytics-events/tree/300

 
Page 1 of 1