Analytics vs. Traditional business App.
Analytics applications has some interesting characteristics that really differs them from our normal business transactional applications.
- It must contain a strong Visualization layer (What is the benefit of doing analysis that is not communicated well to the user).
- It usually deals with multiple and changing data sources so decoupling the data sources from the presentation schema would be of great benefit.
- It does “usually” contain an exploratory phase which adds an interesting aspect to both traditional and modern (Agile) approaches of development.
Proposed Architecture of Open Data Solution.
In my previous posts, the data analytics was done on CSV data direct assuming limited number of users and very limited of resources to build the analytics (volunteer/hackathon type).
But if we start to assume that we have a larger base of users, a faster performing application will be required that can scale well to serve hundreds and thousands of clients per hour, per day.
The proposed Architecture will look like this
This Architecture satisfies the requirements of separating the schema coupled with the Client analytics interface from the Data processing in the backend.
The schema provides a very responsive Solution that can scale very well either on a cloud or on-premise hosted solution.
Adding external real-time resources
The solution could be enhanced further by adding other sources of data that does not required ‘batch’ processing such as social network sentiment analysis or other available services (Weather, financial services, etc.)
Business use of Open Data
The final variation on the previous design is to allow the use of such solution inside a business (Typically a credit organization in a bank that is looking into small business loan).
The solution will allow a bank employee to use Open Data to quickly analyze the potential of a new business opportunity and mixes that with the bank own information about the client to build a holistic approach to evaluating the customer request.
One thing in particular really jumps at me from the previous example, is that some of the data regarding previous businesses in the city (that opened and canceled) include some PII data (client name and phone number).
Now Would a bank want to know if the client have been involved in previous licenses that closed after a certain amount of time ?.
The answer is yes, and they probably should know, and he probably must disclose the fact and the data could be retrieved from previous interactions.
Nonetheless other situations specially ones that involve maybe health history combined with Insurance conditions maybe of high concern.
What is not a PII concern in a regular application could quickly turn into a PII nightmare when mixing different sources of data, specially with the prominence of social networking apps and the ability of business to mine such data.