Microservices Traceability
In a distributed application with a lot of microservices, tracing and monitoring are more important then a monolithic application which you can check the log on each instance even if multiple instances are running. For an application that is composed by numeric microservices running in containers in the cloud, checking the logs in each container to diagnose issues is impossible.
As part of the infrastructure services, centralized logging is a must. All services should send the log to a central location, and then all logs are indexed for full-text search. In a vanilla Kubernetes or Openshift cluster, Fluentd will be used to ingest logs from containers, Elasticseach will be used to index the logs, and Kibana will be used as a UI to view/search/filter the logs. It is called EFK stack in the ecosystem.
For microservices architecture, it is crucial to trace request from one service to another in the entire call tree to have a big picture if something happens or have an audit log that is aggregated by database or Splunk.
In the framework, we have two Ids to serve this purpose.
X-Traceability-Id
- Generated by the original client
- Unique for this client/application only
- Can be database sequence number or a unique identifier like transactionId, orderId, etc.
- Must be passed to the next service
- Must be returned to the caller
- Will be logged in per request audit log if audit handler is enabled
X-Correlation-Id
- Generated in the immediate service from client
- Transparent to the original client
- Must be UUID
- Must be passed to the next service
- Will be logged in per request audit log if audit handler is enabled
- Every service/API must check if this id available and generate one if doesn’t exist in request header.
- It is in the slf4j MDC and in all logging statements
Pass Ids to the next service/API
In order to pass these ids to the next service, the call to the next service must use Client module provided by the framework. It will put these ids to the HttpRequest header with method calls.
Here is the method with its dependency in Http2Client to set JWT tokens, traceabilityId and correlationId.
/**
* Support API to API calls with scope token. The token is the original token from consumer and
* the client credentials token of caller API is added from cache.
*
* This method is used in API to API call
*
* @param request the http request
* @param exchange the http server exchange
*/
public Result propagateHeaders(ClientRequest request, final HttpServerExchange exchange) {
String tid = exchange.getRequestHeaders().getFirst(HttpStringConstants.TRACEABILITY_ID);
String token = exchange.getRequestHeaders().getFirst(Headers.AUTHORIZATION);
String cid = exchange.getRequestHeaders().getFirst(HttpStringConstants.CORRELATION_ID);
return populateHeader(request, token, cid, tid);
}
/**
* Support API to API calls with scope token. The token is the original token from consumer and
* the client credentials token of caller API is added from cache. authToken, correlationId and
* traceabilityId are passed in as strings.
*
* This method is used in API to API call
*
* @param request the http request
* @param authToken the authorization token
* @param correlationId the correlation id
* @param traceabilityId the traceability id
* @return Result when fail to get jwt, it will return a Status.
*/
public Result populateHeader(ClientRequest request, String authToken, String correlationId, String traceabilityId) {
if(traceabilityId != null) {
addAuthTokenTrace(request, authToken, traceabilityId);
} else {
addAuthToken(request, authToken);
}
Result<Jwt> result = tokenManager.getJwt(request);
if(result.isFailure()) { return Failure.of(result.getError()); }
request.getRequestHeaders().put(HttpStringConstants.CORRELATION_ID, correlationId);
request.getRequestHeaders().put(HttpStringConstants.SCOPE_TOKEN, "Bearer " + result.getResult().getJwt());
return result;
}