In this Document describe and illustrate the APIM Statistic publishing and generating model. In this Document, it describes the internal component of the APIM, External Analyzer information and other data retrieval components.

APIM generate an event based on the APIM invocation pattern and published to all the listening event analyzers. The analyzer is responsible to accumulate all the event and generate Summary data based on the defined summarization logic(cApp). After the summarized data are generated APIM Dashboard can retrieve statistics from the Analyzer data-source to UI via APIM analytics client.

a more descriptive component is illustrated in the below diagram.

Internal APIM component listening to the APIM invocations and it’s behavior. Based on the request and responses it generated event and published to all the event receivers. This publisher publish following event streams,

• org.wso2.apimgt.statistics.request

• org.wso2.apimgt.statistics.response

• org.wso2.apimgt.statistics.fault

• org.wso2.apimgt.statistics.throttle

this will publish event stream org_wso2_apimgt_statistics_destination.

APIM component will publish event stream org.wso2.apimgt.statistics.workflow containing work-flow related event data.

The client which is responsible to retrieve statistics data from the available data source. There can be more than one usage client in APIM. In default, there is 2 usage client and either one can use at a time. APIM 1.10.x is composed of two usage client REST and RDBMS client. A user can configure to use either client. By default, REST client is configured and it uses DAS rest API to fetch summarized data.

In the diagram, you can find the path having red and black color. In black color, it demonstrates the path when APIM configure with REST client. In red color, it demonstrates the RDBMS client data flow path. In that case, it stores the same identical data store in the DAS internal table into external RDBMS. These two clients use different summary data generating plan defined in the API_Manager_Analytics_RDBMS.car and API_Manager_Analytics_REST.car.

Summary data generator based on the received event. APIM 1.10.x use DAS 3.0.x to perform analytics task. DAS use apache spark as a processing language. APIM have a pluggable component named DAS Capp which can be deployed into DAS In that cApp contain the all the logic to summarized data and it’s the destination. APIM has two inbuilt capp API_Manager_Analytics_RDBMS.car and API_Manager_Analytics_REST.car. Both contain the same summarization logic except rest client final destination is DAS internal database and an RDBMS it is external RDBMS.

APIM 1.10.0 provide 6 types of the event streams as below.

1. org.wso2.apimgt.statistics.request

   this stream track the information for the API request.

   consumerKey     :       STRING
   context         :       STRING
   api_version     :       STRING
   api             :       STRING
   resourcePath    :       STRING
   method          :       STRING
   version         :       STRING
   request         :       INT
   requestTime     :       LONG
   userId          :       STRING
   tenantDomain    :       STRING
   host            :       STRING
   apiPublisher    :       STRING
   application     :       STRING
   applicationId   :       STRING
   userAgent       :       STRING
   tier            :       STRING
   throttledOut    :       BOOL
   

2. org.wso2.apimgt.statistics.response

   this stream track the information for the API response. It includes the time taken to response get back and received time etc.

   consumerKey     :       STRING
   context         :       STRING
   api_version     :       STRING
   api             :       STRING
   resourcePath    :       STRING
   method          :       STRING
   version         :       STRING
   response        :       INT
   responseTime    :       LONG
   serviceTime     :       LONG
   backendTime     :       LONG
   user            :       STRING
   eventTime       :       LONG
   tenantDomain    :       STRING
   host            :       STRING
   apiPublisher    :       STRING
   application     :       STRING
   applicationId   :       STRING
   cacheHit        :       BOOL
   responseSize    :       LONG
   protocol        :       STRING
     

3. org.wso2.apimgt.statistics.fault

   contain the fault API invocations. It includes the API with backend error, timeout etc

   consumerKey     :       STRING
   context         :       STRING
   api_version     :       STRING
   api             :       STRING
   resourcePath    :       STRING
   method          :       STRING
   version         :       STRING
   errorCode       :       STRING
   errorMessage    :       STRING
   requestTime     :       LONG
   userId          :       STRING
   tenantDomain    :       STRING
   host            :       STRING
   apiPublisher    :       STRING
   application     :       STRING
   applicationId   :       STRING
   protocol        :       STRING
     

4. org_wso2_apimgt_statistics_destination

   this related information based on the endpoint URL

   tenant_id           :       INT
   http_method         :       STRING
   character_set_encoding:       STRING
   remote_address      :       STRING
   transport_in_url    :       STRING
   message_            :       STRING
   remote_host         :       STRING
   service_prefix      :       STRING
   host                :       STRING
   message_direction   :       STRING
   operation_          :       STRING
   message_id          :       STRING
   timestamp           :       LONG
   api                 :       STRING
   version             :       STRING
   request             :       INT
   apiPublisher        :       STRING
   context             :       STRING
   destination         :       STRING
   requestTime         :       LONG
   host                :       STRING
     

5. org.wso2.apimgt.statistics.throttle

   it contains the API invocation with throttle info, throttling can happen due to the following reason.

   • application limit exceed
   • resource limit exceed
   • API limit exceeded
   • hard level limit exceed

   information based on the above throttle is conform with above event

   accessToken         :       STRING
   userId              :       STRING
   tenantDomain        :       STRING
   api                 :       STRING
   api_version         :       STRING
   context             :       STRING
   apiPublisher        :       STRING
   throttledTime       :       LONG
   application         :       STRING
   applicationId       :       STRING
   throttledOutReason  :       STRING
     

6. org.wso2.apimgt.statistics.workflow

   create an event based on the APIM work-flow and publish to the analyzer

   workflowReference   :       STRING
   workflowStatus      :       STRING
   tenantDomain        :       STRING
   workflow            :       STRING
   createdTime         :       LONG
   updatedTime         :       LONG
     

Note that, These Summarized tables are stored in DAS internal storage. Both two cApps APIM deploy on DAS, will first store summary data into these Tables. There is an additional column in some of these tables which contain the composition column of other columns. Those columns type are facet type and which is used to support aggregation function on DAS REST API. Also, all the columns are indexed in order to search using Lucene and support DAS REST API. When APIM configured with RDBMS client all of these tables are replicated in external RDBMS removing facet columns.

• API_REQUEST_SUMMARY

  contain the summary data of the request event stream.

  API_REQUEST_SUMMARY table schema

  CREATE TEMPORARY TABLE API_REQUEST_SUMMARY_FINAL USING CarbonAnalytics OPTIONS (tableName "API_REQUEST_SUMMARY",
    schema "api string -i,
    api_version string -i,
    version string -i,
    apiPublisher string -i,
    consumerKey string -i,
    userId string -i,
    context string -i,
    max_request_time long -i,
    total_request_count int -i,
    hostName string -i,
    year int -i,
    month int -i,
    day int -i,
    time string -i,
    key_api_facet facet -i,
    key_userId_facet facet -i,
    api_version_userId_facet facet -i,
    api_version_userId_apiPublisher_facet facet -i,
    api_version_userId_context_facet facet -i",
    primaryKeys "api,api_version,version,apiPublisher,consumerKey,userId,context,hostName,year,month,day"
    );
  

• API_VERSION_USAGE_SUMMARY

  contain the summary data for APIM usage vise usage. it is also derived from the request event table

  API_VERSION_USAGE_SUMMARY table schema

  CREATE TEMPORARY TABLE API_VERSION_USAGE_SUMMARY_FINAL USING CarbonAnalytics OPTIONS (tableName "API_VERSION_USAGE_SUMMARY",
    schema "api string -i,
    version string -i,
    apiPublisher string -i,
    context string -i,
    total_request_count int -i,
    hostName string -i,
    year int -i,
    month int -i,
    day int -i,
    time string -i,
    max_request_time long -i,
    api_version_context_facet facet -i",
    primaryKeys "api,version,apiPublisher,context,hostName,year,month,day"
    );
  

• API_Resource_USAGE_SUMMARY

  contain the summarized data for APIM usage by resources and it is also derived from the request event table

  API_Resource_USAGE_SUMMARY table schema

    CREATE TEMPORARY TABLE API_Resource_USAGE_SUMMARY_FINAL USING CarbonAnalytics OPTIONS (tableName "API_Resource_USAGE_SUMMARY",
    schema "api string -i,
    version string -i,
    apiPublisher string -i,
    consumerKey string -i,
    resourcePath string -i,
    context string -i,
    method string -i,
    total_request_count int -i,
    hostName string -i,
    year int -i,
    month int -i,
    day int -i,
    time string -i,
    max_request_time long -i,
    key_api_method_path_facet facet -i,
    api_version_context_method_facet facet -i",
    primaryKeys "api,version,apiPublisher,consumerKey,context,resourcePath,method,hostName,year,month,day"
    );
  

• API_RESPONSE_SUMMARY

  contain the summarized data from API responses. it is derived from the response event table

  API_RESPONSE_SUMMARY table schema

    CREATE TEMPORARY TABLE API_RESPONSE_SUMMARY_FINAL USING CarbonAnalytics OPTIONS (tableName "API_RESPONSE_SUMMARY",
    schema "api_version string -i,
    apiPublisher string -i,
    context string -i,
    serviceTime int -i,
    total_response_count int -i,
    hostName string -i,
    year int -i,
    month int -i,
    day int -i,
    time string -i,
    max_request_time long -i,
    api_version_context_facet facet -i",
    primaryKeys "api,version,apiPublisher,context,hostName,year,month,day"
    );
  

• API_FAULT_SUMMARY

  contain the summarized data of faulty API invocation. it is derived from the fault event stream

  API_FAULT_SUMMARY table schema

    CREATE TEMPORARY TABLE API_FAULT_SUMMARY_FINAL USING CarbonAnalytics OPTIONS (tableName "API_FAULT_SUMMARY",
    schema "api string -i,
    version string -i,
    apiPublisher string -i,
    consumerKey string -i,
    context string -i,
    total_fault_count int -i,
    hostName string -i,
    year int -i,
    month int -i,
    day int -i,
    time string -i,
    max_request_time long -i,
    consumerKey_api_facet facet -i,
    api_version_apiPublisher_context_facet facet -i",
    primaryKeys "api,version,apiPublisher,consumerKey,context,hostName,year,month,day"
    );
  

• API_DESTINATION_SUMMARY

  contain the summarized data of API destination. and it is derived from the destination event stream

  API_DESTINATION_SUMMARY table schema

    CREATE TEMPORARY TABLE API_DESTINATION_SUMMARY_FINAL USING CarbonAnalytics OPTIONS (tableName "API_DESTINATION_SUMMARY",
    schema "api string -i,
    version string -i,
    apiPublisher string -i,
    context string -i,
    destination string -i,
    total_request_count int -i,
    hostName string -i,
    year int -i,
    month int -i,
    day int -i,
    time string -i,
    max_request_time long -i,
    api_version_context_dest_facet facet -i",
    primaryKeys "api,version,apiPublisher,context,destination,hostName,year,month,day"
    );
  

• API_LAST_ACCESS_TIME_SUMMARY

  contain the summary data of the last access time of the API, it is also derived from the request event stream

  API_LAST_ACCESS_TIME_SUMMARY table schema

    CREATE TEMPORARY TABLE API_LAST_ACCESS_TIME_SUMMARY_FINAL USING CarbonAnalytics OPTIONS (tableName "API_LAST_ACCESS_TIME_SUMMARY",
    schema "tenantDomain string -i,
    apiPublisher string -i,
    api string -i,
    version string -i,
    userId string -i,
    context string -i,
    max_request_time long -i",
    primaryKeys "tenantDomain,apiPublisher,api"
    );
  

• API_THROTTLED_OUT_SUMMARY

  contain the summary of the throttle out API invocation data. it is derived from the throttle out event table and request table

  API_THROTTLED_OUT_SUMMARY table schema

  CREATE TEMPORARY TABLE THROTTLED_OUT_FINAL_SUMMARY USING CarbonAnalytics OPTIONS (tableName "API_THROTTLED_OUT_SUMMARY",
    schema "api string -i,
    api_version string -i,
    context string -i,
    apiPublisher string -i,
    applicationName string -i,
    tenantDomain string -i,
    year int -i,
    month int -i,
    day int -i,
    week int -i,
    time string -i,
    success_request_count int -i,
    throttleout_count int -i,
    max_request_time long -i,
    api_year_month_week_day_facet facet -i,
    applicationName_facet facet -i",
    primaryKeys "api,api_version,context,apiPublisher,applicationName,tenantDomain,year,month,day"
  );
  

• Publisher

  In Publisher there are two categories All statistic and statistic section. both contain the following types of statistics. In all statistics section, you can see the statistics for the current tenant domain. For the statistics section, you can see the data for the current logging publisher related information.

  • API Subscriptions

    list of APIs with its current subscriptions

    

  • API Usage

    List of APIs with it’s hit count

    

  • API Response Times

    List APIs with its average response time

    

  • API Last Access Times

    List of APIs with it’s last access time

    

  • API Usage by Resource Path

    List of APIs with it’s hit count for each resource and methods

    

  • API Usage by Destination

    API endpoint hit counts

    

  • API Usage Comparison

    API hits counts and hit count comparison with all other APIs

    

  • API Throttled Out Requests

    API success and throttle out count by APIs and application

    

  • App Throttled Out Requests

    API success and throttle out count by application

    

  • Faulty Invocations

    API fault count and its percentage with respect to all request count

    

• Store
  • API Usage per Application

    API usage by application wise

    

  • Top Users per Application

    top users based on the application

    

  • API Usage from Resource Path per Application

    API resources for API and application

    

  • Faulty Invocations per Application

    Faulty API invocation by application wise