Showing posts with label data warehousing and mining. Show all posts
What is cryptography
cryptography
If you use the internet, you will hear these words very often and you will have seen a feature called encryption in WhatsApp too. But maybe you do not know what it is, so do not worry, by the end of this audio lesson you will know about this topic.
In the world of the Internet, the term Encryption and Decryption is very popular. Internet is a wide network where nothing is secure, so to protect data in internet encryption technique is used. if you do not implement encryption so Anyone can hack your data and use it for the wrong things.
The process of making the message (or data) unusable or unreadable for all in which only the recipient can understand it is called Encryption. The process (or science) of creating procedures useful for encrypting and decrypting messages is called cryptography. The purpose of this process is only that the message is not decrypted (identified / read) by the suspected user insufficient time.
So what is Cryptography? The word Cryptography is derived from the Greek language word Krypto, which means "Hidden Secrets". Or in other words, it is an art in which we can protect our data and information, through Cryptography we can convert our data into Unreadable Secret Codes which are called Cipher Text, and can read this data the same. Who will have a secret key to decrypt it. Decrypt data is called plain text. There are two processes Encryption and Decryption. It is used to protect E-Mail Messages, Credit / Debit Card details, and other important information. In Encryption, Plain Text is converted to Cipher Text and in Decryption Cipher Text is converted to Plain Text. <Div class = "code-block
How does this work?
So guys when you send a message using an encrypted messaging service (for example WhatsApp), it converts the service message into code, scrambles it, and creates an encryption key. After this, only the right recipient can unlock this message. Digital encryption is extremely complex and that is why it is considered difficult to crack. To strengthen its security, every time two smartphones start communicating with each other, a new set of encryption algorithms is created.
Cryptography has two types
First Symmetric Cryptography in which the keys of Encryption and Decryption are the same means that the data or information is Encrypt and Decrypt using the same key, Second Asymmetric Cryptography in which Keys of both Encryption and Decryption are different means that Encryption in it Public key is used to do this and Private key is used to do the decryption.
Cryptography has many advantages because encryption makes your data completely secure and safe, even if it is hacked or stolen after encrypting the data, no one can access or read your data. And using encryption, your data can only be accessed by anyone who wants to allow access to it or a password will be required.
big data analytics
big data analytics
Contents
- What is big data
- big data analytics
- benefits of big data analytics
- types of big data analytics
what is big data
The term 'Big data' is used to describe a large amount of data.
Simply put, "big data means huge data". And this data goes on growing with time.
This data is so large and complex that it is very difficult to store and process through a traditional software application.
Here's an example to understand big data: -
Every day more than 500 terabytes of data are generated in the database of Facebook. This data is mainly generated from photos and video uploads, messages, comments, etc.
big data analytics
"Big data analytics is a process in which a large group of data is collected, organized and analyzed so that hidden patterns and useful information can be discovered."
In other words, "big data analytics is a process in which large data sets are examined so that organizations can get hidden patterns, market trends, customer preferences, and other useful information."
This information is used by the organization to improve its business decisions.
Data scientists and predictive modelers, through big data analytics, analyze data from very summary sources.
benefits of big data analytics
The benefits of big data analytics are as follows: -
1: - By doing so, the company or organization can make a better decision, i.e. it can make a better decision by accessing data from search engines and social media sites like - Facebook, Twitter, etc.
2: - By this, the error of the company is detected very quickly. With the help of real-time insights into the errors, the company quickly resolves the problem.
3: - This is better customer service. When a company monitors the product used by the customer, it remains ready for any future failure.
For example - Cars that contain real-time sensors, before the sensors crash, they tell the driver that there is some disturbance in the car.
4: - cost savings: - The cost of implementing big data tools can be very high. But these tools save a lot of money and these companies are very beneficial for the company. Through them, we can store large amounts of data. And these tools also identify effective ways of business.
5: - It saves time. Big data tools such as Hadoop and in-memory analytics are much faster. These tools easily identify new sources of data, so that data is analyzed very quickly and early decisions are taken based on learning.
6: - new product development: -Customers are using which product they are using and what they have needs are addressed by big data analytics. So based on these analytics, we can develop a new product according to the needs of the customer.
7: By this, we can understand the condition of the market, after analyzing the big data, we know what is the condition of the market. For example: - If the company finds out what the customers in the market Buying and selling which product is the most, the company will be one step ahead of its competitors.
8: - By this, the company can control its online reputation. Through big data tools, the company knows what the customers are giving feedback about the company.
If the company wants to monitor and improve its online reputation, it can do with the help of big data analytics and tools.
9: - This leads to fraud. The criminals do fraud online now, but if someone hacks the system of a criminal or hacker company, then the company gets its address immediately and the company's IT department can immediately take necessary action.
The following types of big data analytics are: -
1: - perspective analytics
2: - predictive analytics
3: - diagnostic analytics
4: - descriptive analytics
1. Perspective analytics
This is the most valuable big data analytics technique, it suggests the best solution among many choices. So that the suggested option can be availed. And future risks can be reduced.
2.Predictive analytics
The predictive analytics is most commonly used. It predicts something that can happen in that situation?
It uses statistical, data modeling, data mining, and machine learning techniques to predict the situation.
3.Diagnostic analytics
Data scientists use this technique when they want to know why something is happening, i.e. what is the reason behind this thing?
diagnostic analytics which analyzes the past performance
4.Descriptive analytics
This technique takes a lot of time and gives the least benefit.
What is descriptive analytics is that it provides an insight into the data such as: - summary statistics,
clustering and association rules etc.
data warehouse data warehousing features of datawarehouse
data warehouse data warehousing features of data warehouse
The term “data warehouse” was first coined by Bill Inmon in the year 1990.
According to Inmon, a data warehouse is a subject-oriented, integrated, time-variant, and non-volatile collection of data.
A data warehouse is a collection of multiple databases.
A data warehouse provides us generalized and consolidated data in the multidimensional view.
Along with a generalized and consolidated view of data.
A data warehouse also provides us online analytical processing (OLAP).
·Understanding the data warehouse:-
1.A data warehouse is a database, which is kept separate from the organization’s operational database.
2.There is no frequent updating done in a data warehouse.
A data warehouse is connected by integrating data from multiple heterogeneous sources that support analytical reporting, structured Adhoc queries and decision making.
Data warehousing
Data warehousing means of storing data in one place. In this medium, many types of hardware are used to store the data. Store data on a group or computer of servers is called data warehousing. New and historical data is stored in the data warehouse.
The function of data warehousing is not only to store data but it also provides data protection.
A data warehouse collects information from many types of resources in one place that is called a repository. All information in it is stored in the same schema. Once the collected data stays in the store and it can be accessed for a long time.
features of data warehouse: -
1: -Subject oriented: -Data warehouse is used to analyze a particular subject, that is, it is based on the particular subject.
2: -Integrated: -data warehouse collects data from sources of data in very summary.
3: -Time variant: that means new and historical data is kept in it. Data at any time can be kept in the data warehouse.
4: -Non-volatile: - Once the data is stored in the data warehouse, that data can not be changed, meaning the historical data can not be changed.
data warehouse architecture
data warehouse architecture
Data warehouse architecture is a design that encapsulates aspects of data warehousing.
Through the data warehousing architecture, we can maintain any objective. The data warehouse architecture is made by keeping in mind the needs of the client and the goal of the employee.
Many sources of data warehouses have different types of architecture. Here we will read about the data warehouse's three-tier architecture, which is the most used.
data warehouse architecture
three tier data warehouse architecture
There are three layers in the data warehouse architecture.
1: -bottom tier (lower layer)
2: -middle tier (middle layer)
3: -top tier (highest layer)
data warehouse architecture
data warehouse architecture
1: - bottom tier: -
The bottom tier of the architecture is the data warehouse database.
It is the relational database system.
These back and tools and utilities perform the extract, clean, load and refresh.
This layer handles the data warehouse's ETL process. The ETL process is the process in which data is extracted, transform, and load in the data warehouse.
The ETL (extract, transform, and load) process is performed by back end tools.
2: - middle tier: -
This tier is also called the application tier. This layer handles the request of all users. BI (Business Intelligence) is built in this layer.
By multidimensional OLAP (MOLAP) model, this directly implements the multidimensional data and operations.
This layer contains an OLAP server. This OLAP server is implemented in the following two types.
i: - ROLAP (Relational OLAP): - These servers are an intermediate server that is located between the relational back end server and the client front end tools.
ii: - MOLAP (multidimensional OLAP): - This server supports the multi-dimensional view of data.
3: - top tier: - This tier is also called the presentation tier. This front end is the client layer. This layer includes query tools, reporting tools, analysis tools and
There are data mining tools. This layer interacts with end users. That is, by this layer, the data reaches the end users.
data warehouse architecture
data warehouse architecture
benefits and problems of data warehousing
benefits and problems of data warehousing
benefits and problems of data warehousing
benefits of data warehouse
The advantages of data warehousing are as follows: -
1: - The biggest advantage of data warehousing is that users who have access to the source of large information can help solve many problems.
2: -It increases business intelligence.
3: -The system's performance and queries increase.
4: -We can access data on time.
5: -The quality of the data increases and the continuity increases.
6: -With more than we can afford, we can earn more from it.
7: -Companies make good and smart decisions by this
8: -It supports ad hoc reporting and enquiry.
9: -We can access data quickly and easily.
benefits and problems of data warehousing
problems of data warehouse
The following are the problems in the data warehouse.
1: - Sometimes it takes longer to sort, clear, and load the data in the warehouse. Which is one of the biggest problems of data warehouses.
2: - Data Warehouse is a system that needs high quality maintenance and the cost of this maintenance is very high.
3: - Data comes from many organizational and sources in Data Warehouse. In which a lot of data is inconsistent and duplicate, which reduces the quality of data which is a major problem.
4: -In conceptual schema for documenting the project is not enough. This requires multiline and integrated approaches.
5: Testing in Data Warehousing is like a big challenge, it takes a lot of time to do testing in such a large database.
benefits and problems of data warehousing
Problems with Data WareHousing
i).Data quality:-In a data warehouse, data is coming from many disparate sources from all facts of an organization.
When a data warehouses tries to combine inconsistent data from disparate sources, it encounters error.
ii).Designing the data warehouse:-People generally don’t want to “waste” their time defining the requirements necessary for proper data warehouse design.
iii).Testing the data warehouse:-Testing in data warehousing is a real challenges.
It takes lot of time to test them.
One of the reasons why testing is trickily is due to the reason that a top level object in data warehouse.
iv).Under estimation of resources of data loading:-Sometimes we under estimate the tie required extracting, clean and loading the data into the warehouse.
benefits and problems of data warehousing
v).Hidden problems with source systems:-sometimes hidden problems associated with the source systems feeding the data warehouse may be indentified of ten years of being untested.
vi).Required data not captured:-In some cases, the required data is not captured by the source systems which may be very important for the data warehouse purpose. For example-the data of registration of the property may be not used in source system but it may be very important for analysis purpose.
vii).Increased end-user demands:-After satisfying some of end users queries, requests for support from staff may increase rather than decrease. This is caused by an increasing awareness of the users on the capability and value of the data warehouse.
viii).High demand for resources:-The data warehouse requires large amounts of dat.
ix). Complexity of integration:-The most important area for the management of data warehouse is the integration capabilities. This can be very difficult task, as there are a number of tools for every operation of the data warehouse.
x).High maintenance:-Data warehouses are high maintenance systems. Any reorganization of the business processes and the source systems may affect the data warehouse and it results in high maintenance.
online analytical processing (olap) and online transaction processing (oltp) and types of OLAP
online analytical processing (olap) and online transaction processing (oltp) and types of OLAP
olap:- The full name of olap is Online Analytical Processing
Online analytical processing is database organisation process in which large business databases organized and support intelligence.
olap is online database querying system.
olap is what works with historical data. Data in olap is not normalized, meaning data is redundant.
olap is used to analyze the data and the data remains stored in the multi-dimensional schema. The data in it is non-volatile.
online analytical processing (olap) and online transaction processing (oltp) and types of OLAP
Types of olap:-
1) Relational olap(ROLAP)
2) Multi dimensional olap(MOLAP)
3) Hybrid olap(HOLAP)
4) Specialized SQL servers
online analytical processing (olap) and online transaction processing (oltp) and types of OLAP
1. Relational olap:-To store and manage warehouse data.
2.Multi dimensional olap:-MOLAP uses based multi dimensional storage engines for multi dimensional view of data.
3. Hybrid olap(HOLAP):-Hybrid olapis a combination of both ROLAP and OLAP.
4. Specialized SQL servers:-SQl stands for structured query language so sql servers are used to provide data processing by query processing with the use of sql queries.
oltp: Online Transaction Processing
oltp is a class of information system that facilitates and manages transaction oriented applications, typically for data entry and retrieval transaction processing on a management system.
It works with original ie operational data and oltp processes the transaction for each day.
oltp is online database modifying system.
online analytical processing (olap) and online transaction processing (oltp) and types of OLAP
For example: - Whatever process is taken out of the bank, such as: - inserting atm card, inserting PIN code, etc., comes inside oltp.
It is used to store data at a faster speed. In it the volatile data and current data are executed and there is no historical data and oltp has a detailed data store.
online analytical processing (olap) and online transaction processing (oltp) and types of OLAP
Online analytical processing is database organisation process in which large business databases organized and support intelligence.
olap is online database querying system.
olap is what works with historical data. Data in olap is not normalized, meaning data is redundant.
olap is used to analyze the data and the data remains stored in the multi-dimensional schema. The data in it is non-volatile.
online analytical processing (olap) and online transaction processing (oltp) and types of OLAP
- It is based on the multi-dimensional data model that allows used to analyze database information from multiple database systems at one time.
- It allows users and analysts to get information through fast consistent and interactive access to information.
- olap applies consolidation data; olap data comes from the various olap databases.
- olap is a class of systems, which provide answers to multi dimensional quarries.
- olap tools enable users to analyze multidimensional and data interactively from multiple perspectives. olap consist of three basic analytical operations.
- Consolidation (roll-up), drill-down, slicing and dicing.
- olap is computer processing a user to easily and selectively extract and view data from different points of view.
- olap deals with historical data or archival data. olap is charactererized by relatively low volume of .
online analytical processing (olap) and online transaction processing (oltp) and types of OLAP
Types of olap:-
1) Relational olap(ROLAP)
2) Multi dimensional olap(MOLAP)
3) Hybrid olap(HOLAP)
4) Specialized SQL servers
online analytical processing (olap) and online transaction processing (oltp) and types of OLAP
1. Relational olap:-To store and manage warehouse data.
2.Multi dimensional olap:-MOLAP uses based multi dimensional storage engines for multi dimensional view of data.
3. Hybrid olap(HOLAP):-Hybrid olapis a combination of both ROLAP and OLAP.
4. Specialized SQL servers:-SQl stands for structured query language so sql servers are used to provide data processing by query processing with the use of sql queries.
oltp: Online Transaction Processing
oltp is a class of information system that facilitates and manages transaction oriented applications, typically for data entry and retrieval transaction processing on a management system.
It works with original ie operational data and oltp processes the transaction for each day.
oltp is online database modifying system.
online analytical processing (olap) and online transaction processing (oltp) and types of OLAP
For example: - Whatever process is taken out of the bank, such as: - inserting atm card, inserting PIN code, etc., comes inside oltp.
It is used to store data at a faster speed. In it the volatile data and current data are executed and there is no historical data and oltp has a detailed data store.
online analytical processing (olap) and online transaction processing (oltp) and types of OLAP
- Data in the oltp database is normalized and this means that the data is not redundant (extra).
- oltp is characterized by a large of short on-line transaction (INSERT, UPDATE, and DELETE).
- The main emphasis for oltp systems is fast query processing , access environment and an effectiveness measured by number of transactions per second.
- In oltp database there is a detailed and current data and schema used to store transactional database is the entity model. It involves ovaries accessing individual record like update your Email in company database.
olap vs oltp
olap vs oltp
OLAP
|
OLTP
|
olap involves
historical processing of information.
|
oltp involves
day-today processing.
|
olap is used to
analyze the business.
|
oltp is used to
run business.
|
olap focuses on
information out.
|
oltp focuses on
data in.
|
olap contains
historical data.
|
oltp contains
current data.
|
olaps are highly
flexible.
|
oltp provides high
performance.
|
100
MB to 100 GB.
|
100
GB to 100 TB.
|
Data
model:-Multi dimensional operates data.
|
Normalized
|
Source
of data:-oltp are the
source of olap data.
First
data comes in oltp and then olap
|
oltp are the
original source of data.
|
Processing
speed:-Depends
upon the amount of data involved.
|
Typically
fast.
|
Quarries:-Often complex
queries involving aggregations.
|
Relatively
standardized and simple queries.
|
Insert
and updates:-periodic
long running batch jobs refresh the data.
|
Fast
insert , updates done by end user itself
|
Database
design:-Typically
denormalized with fewer tables.
|
Highly
normalized with many tables.
|
cluster analysis in data mining
cluster analysis in data mining
cluster analysis in data mining is the classification of objects into different groups or the portioning of dataset into subsets (cluster).
In other words, similar objects are grouped in one cluster and dissimilar objects are grouped in another duster.
While doing cluster analysis, we first partition the set of data into groups based on data similarly and then assign the lables to the groups.
Cluster analysis is employed for the analysis of production flow analysis chart to determine feasible groups of processes and their respective packs of parts. It makes use of algorithms for the study of similarties between objects in a quantitative manner as compared to the classification techniques, which appears to be descriptive.
We may define cluster analysis in data mining as the science of the classification of objects based on their possession or lack of defined characteristics.
This technique shows an approach to study the similarities between a diverse population of objects in a quantitative manner
cluster analysis in data mining methods:-
- Partioning method
- Hierarchical method
- Density based method
- Grid based method
- Model based method
- Constraint based method
1. Partioning method:-
Suppose we are given a database of ‘n’ objects and the portioning method constructs ‘k’ partion of data.
Each portion will represent a cluster and (k≤n)
Each group contains at least one object.
Each object must belong to exactly one group.
2. Hierarchical methods:-
As its name suggests a hierarchical tree of given set of data object is created in this method
We can classify hierarchical methods on the basis of how the hierarchical decomsition is formed.
There are two approaches
- Agglomerative approach
- Divisive approach
cluster analysis in data mining
a. Agglomerative approach:-
Agglomerative approach also suggests as bottom up approach.This process start with separating each objects into groups.
b. Divisive approach:-
Divisive approach also called as top down approach.
All the objects are grouped into same cluster.
This method is rigid, i.e. once a merging or splitting is done, and it can never be undone.
3. Density based method:-The basic idea is to continue growing. The given cluster as long as the density in the heighbourhood exceeds some threshold i.e. for each data point within a given cluster, the radius of given cluster has to contain at least a minimum number of points.
4. Grid based method:- The objects is quantized into finite number of cells that forms grid structure.
Advantages:-Fast processing time.
5. Model based method:-
This methods locates the cluster by clustering the density function.
This method also provides a way to automatically determine the number of clusters based on standard statics, taking out lier or noise into account.
6. Constraint based method :-In this method, the clustering is performed by the incorporation of user or application oriented constraints.
A constraint refer to the user expectation or the properties of desired clustering results.
Constraint can be specified by the user for the application requirement.
=>. Three stages of cluster analysis in data mining
Clustering consists of the following three stages:
Preparing a post-operation matrix. This shows whether certain features (like a keyway on shaft) are present or absent.
Computing a similarity co-efficient matrix. The bases of this are the extent to which the parts share common characteristics. In this case, co-efficient would have a value of one (1) when parts are identical and ten (10) when they have no common entity.
Performing a clustering analysis. In this case, the similarity between each pair of objects is examined and group of objects formed such that, within each group, the objects are similar to each other according to the set of rules which have been formulated previously.
data mining problems and issues
data mining problems and issues
Data mining is not an easy task, as the algorithm used can get very complete and data is not always available at one place.
It needs to be integrated from various hetrogenous data sources.
The major issues in data mining process are as the following:
- Mining methodology and user interaction
- Performance issues
- Diverse data types issues
- User interaction
- Data types
- Huge size of database
- Wide distribution of data
- High cost
- Individual privacy
- Data integrity
Some genuine and real data mining problems and issues are described below:-
data mining issues (problems)
The area of data mining is very large and its work is not easy. Every day, there is a large amount of data generation and store. So that the complexity of data mining is going on increasing and due to this there are issues and problems too. Let's read below that they have data mining issues .
data mining problems and issues
1: - The first issue of data mining is cost. because the queries of data mining are very complex, due to which data has to be collected and maintained in the data, and for all of us, we need large and fast systems which are very expensive.
2: - Handling unstable, uncompromising, and cost sensitive data is a major problem of data mining.
data mining problems and issues
3: - Another issue of data mining is the redundancy. It means duplication of lots of data of the same type of data.The same data is copied so that the original data is found and separated from them.
4: - The issue of privacy and security is very big. Everyone has to maintain the privacy and security of the organization and the organization.
5: - Managing bad data quality i.e. managing incomplete, unstable, noisy and high dimensional data.
6: - Trouble missing any data or access to data
data mining problems and issues
7: - Continuous updating of old data and handling new data.
8: - Processing large, hard, immovable and unorganized data in structured format.
9: - To do data mining, data miners should have background knowledge because this gives us complete information about the patterns of the data and the process. Without this background knowledge, this work is not easy.
10: - Data mining algorithm should be efficient and scalable. If it is not so, then we will neither benefit nor only negative results will come out.
11: - Every person needs different types of knowledge, therefore, data mining has to be discovered by the person as needed.
Subscribe to:
Posts
(
Atom
)