Open access


This paper shows how big data analysis opens a range of research and technological problems and calls for new approaches. We start with defining the essential properties of big data and discussing the main types of data involved. We then survey the dedicated solutions for storing and processing big data, including a data lake, virtual integration, and a polystore architecture. Difficulties in managing data quality and provenance are also highlighted. The characteristics of big data imply also specific requirements and challenges for data mining algorithms, which we address as well. The links with related areas, including data streams and deep learning, are discussed. The common theme that naturally emerges from this characterization is complexity. All in all, we consider it to be the truly defining feature of big data (posing particular research and technological challenges), which ultimately seems to be of greater importance than the sheer data volume.

Ahmadov, A., Thiele, M., Eberius, J., Lehner, W. and Wrembel, R. (2015). Towards a hybrid imputation approach using web tables, IEEE/ACM International Symposium on Big Data Computing (BDC), Limassol, Cyprus, pp. 21-30.

Bekkerman, R., Bilenko, M. and Langford, J. (2011). Scaling Up Machine Learning: Parallel and Distributed Approaches, Cambridge University Press, New York, NY.

Benjelloun, O., Garcia-Molina, H., Menestrina, D., Su, Q., Whang, S.E. and Widom, J. (2009). Swoosh: A generic approach to entity resolution, The VLDB Journal 18(1): 255-276.

Bayer, M.A. and Edjlali, R. (2014). Magic Quadrant for Data Warehouse Database Management Systems, Gartner Publications, Stamford, CT,

Beyer, M. and Laney, D. (2012). The Importance of “Big Data”: A Definition, Gartner Publications, Stamford, CT.

Boyd, D. and Crawford, K. (2012). Critical questions for big data, Information, Communication and Society 15(5): 662-679.

Brzezinski, D. and Stefanowski, J. (2014). Combining block-based and online methods in learning ensembles from concept drifting data streams, Information Sciences 265: 50-67.

Che, D., Safran, M. and Peng, Z. (2013). From big data to big data mining: Challenges, issues, and opportunities, in

B. Hong et al. (Eds.), International Conference on Database Systems for Advanced Applications, Lecture Notes in Computer Science, Vol. 7827, Springer, Berlin/Heidelberg, pp. 1-15.

Chen, C.L.P. and Zhang, C. (2014). Data-intensive applications, challenges, techniques and technologies: A survey on big data, Information Sciences 275(10): 314-347.

Custers, B., Calders, T., Schermer, B. and Zarsky, T.Z. (Eds.) (2013). Discrimination and Privacy in the Information Society-Data Mining and Profiling in Large Databases, Studies in Applied Philosophy, Epistemology and Rational Ethics, Vol. 3, Springer, Berlin/Heidelberg.

Ditzler, G., Roveri, M., Alippi, C. and Polikar, R. (2015). Learning in nonstationary environments: A survey, IEEE Computational Intelligence Magazine 10(4): 12-25.

Domingos, P. and Hulten, G. (2000). Mining high-speed data streams, ACM SIGKDD International Conference on Knowledge Discovery Data Mining, Boston, MA, USA, pp. 71-80.

Duggan, J., Elmore, A.J., Stonebraker, M., Balazinska, M., Howe, B., Kepner, J., Madden, S., Maier, D., Mattson, T. and Zdonik, S. (2015). The BigDAWG polystore system, SIGMOD Record 44(2): 11-16.

Elmagarmid, A., Rusinkiewicz, M. and Sheth, A. (Eds.) (1999). Management of Heterogeneous and Autonomous Database Systems, Morgan Kaufmann, San Francisco, CA.

Fernández, A., del Río, S., Chawla, N.V. and Herrera, F. (2017). An insight into imbalanced big data classification: Outcomes and challenges, Complex & Intelligent Systems 3(2): 105-120.

Francisco, P. (2012). Oracle Exadata and IBM Netezza data warehouse appliance compared, IBM White Paper,

Gama, J. (2010). Knowledge Discovery from Data Streams, Chapman and Hall, Boca Raton, FL.

Gama, J., Zliobaite, I., Bifet, A., Pechenizkiy, M. and Bouchachia, A. (2014). A survey on concept drift adaptation, ACM Computing Surveys 46(4): 44:1-44:37.

Gens, F. (2011). IDC predictions 2012: Competing for 2020. IDC analyze the future,

Gessert, F., Schaarschmidt, M., Wingerath, W., Witt, E., Yoneki, E. and Ritter, N. (2017). Quaestor: Query web caching for database-as-a-service providers, PVLDB 10(12): 1670-1681.

Glavic, B. (2014). Big data provenance: Challenges and implications for benchmarking, in T. Rabl et al. (Eds.), Specifying Big Data Benchmarks, Springer, New York, NY, pp. 72-80.

Gupta, A. (2009). Data provenance, in L. Liu and M.T. O¨ zsu (Eds.), Encyclopedia of Database Systems, Springer, Berlin, pp. 608-608.

Han, J. and Kamber, M. (Eds.) (2011). Data Mining. Concepts and Techniques, Morgan Kaufmann, San Francisco, CA.

Hashem, H. and Ranc, D. (2016). Pre-processing and modeling tools for bigdata, Foundations of Computing and Decision Sciences 41(3): 151-162.

Japkowicz, N. and Stefanowski, J. (2016a). A machine learning perspective on big data analysis, in N. Japkowicz and J.

Stefanowski (Eds.), Big Data Analysis: New Algorithms for a New Society, Springer, Cham, pp. 1-31.

Japkowicz, N. and Stefanowski, J. (Eds.) (2016b). Big Data Analysis: New Algorithms for a New Society, Studies in Big Data, Vol. 16, Springer, Cham.

Kingma, D.P. and Welling, M. (2013). Auto-encoding variational Bayes, ArXiv e-prints, 1312.6114a.

Krawczyk, B., Minku, L.L., Gama, J., Stefanowski, J. and Wozniak, M. (2017). Ensemble learning for data stream analysis: A survey, Information Fusion 37: 132-156.

Krempl, G., Zliobaite, I., Brzezinski, D., H¨ullermeier, E., Last, M., Lemaire, V., Noack, T., Shaker, A., Sievi, S., Spiliopoulou, M. and Stefanowski, J. (2014). Open challenges for data stream mining research, SIGKDD Explorations 16(1): 1-10.

Krizhevsky, A., Sutskever, I. and Hinton, G.E. (2012). Imagenet classification with deep convolutional neural networks, in F. Pereira et al. (Eds.), Advances in Neural Information Processing Systems 25, Curran Associates, Inc., Red Hook, NY, pp. 1097-1105.

Krawiec, K. (2016). Evolutionary feature selection and construction, in S. Claude and G. Webb (Eds.), Encyclopedia of Machine Learning and Data Mining, Springer, Boston, MA.

Langegger, A., Wöß, W. and Blöchl, M. (2008). A semantic web middleware for virtual data integration on the web, European Semantic Web Conference on the Semantic Web: Research and Applications (ESWC), Tenerife, Canary Islands, Spain, pp. 493-507.

LeCun, Y., Bengio, Y. and Hinton, G. (2015). Deep learning, Nature 521(7553): 436-444.

Liu, M. and Wang, Q. (2016). Rogas: A declarative framework for network analytics, International Conference on Very Large Data Bases (VLDB), New Delhi, India, pp. 1561-1564.

Matwin, S. (2013). Privacy-preserving data mining techniques: Survey and challenges, in B. Custers et al. (Eds.), Discrimination and Privacy in the Information Society, Vol 3. Springer, Berlin/Heidelberg, pp. 209-221.

Mauro, A.D., Greco, M. and Grimaldi, M. (2015). What is big data? A consensual definition and a review of key research topics, International Conference on Integrated Information, Madrid, Spain, pp. 97-104.

Miao, X., Gao, Y., Guo, S. and Liu, W. (2017). Incomplete data management: A survey, Frontiers of Computer Science, DOI: 10.1007/s11704-016-6195-x.

Moreau, L., Clifford, B., Freire, J., Futrelle, J., Gil, Y., Groth, P., Kwasnikowska, N., Miles, S., Missier, P., Myers, J., Plale, B., Simmhan, Y., Stephan, E. and den Bussche, J.V. (2011). The open provenance model core specification (v1.1), Future Generation Computer Systems 27(6): 743-756.

Napierala, K. and Stefanowski, J. (2016). Types of minority class examples and their influence on learning classifiers from imbalanced data, Journal of Intelligent Information Systems 46(3): 563-597.

Naumann, F. (2014). Data profiling revisited, SIGMOD Record 42(4): 40-49.

Rudin, C. (2014). Algorithms for interpretable machine learning, ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, New York, NY, USA, pp. 1519-1519.

Russom, P. (2017). Data lakes: Purposes, practices, patterns, and platforms. TDWI White Paper,

Schmidhuber, J. (2015). Deep learning in neural networks: An overview, Neural Networks 61(C): 85-117.

Shaker, A. and Hüllermeier, E. (2014). Survival analysis on data streams: Analyzing temporal events in dynamically changing environments, International Journal of Applied Mathematics and Computer Science 24(1): 199-212, DOI: 10.2478/amcs-2014-0015.

Soltanpoor, R. and Sellis, T. (2016). Prescriptive analytics for big data, Australasian Database Conference on Databases Theory and Applications (ADC), Sydney, Australia, pp. 245-256.

Sun, Y., Tang, K., Minku, L.L., Wang, S. and Yao, X. (2016). Online ensemble learning of data streams with gradually evolved classes, IEEE Transactions on Knowledge and Data Engineering 28(6): 1532-1545.

Terrizzano, I., Schwarz, P., Roth, M. and Colino, J.E. (2015). Data wrangling: The challenging journey from the wild to the lake, Conference on Innovative Data Systems Research (CIDR), Asiloma, CA, USA.

Wang, J., Crawl, D., Purawat, S., Nguyen, M.H. and Altintas, I. (2015). Big data provenance: Challenges, state of the art and opportunities, IEEE International Conference on Big Data, Santa Clara, CA, USA, pp. 2509-2516.

Wiederhold, G. (1992). Mediators in the architecture of future information systems, IEEE Computer 25(3): 38-49.

Wylot, M., Cudré-Mauroux, P., Hauswirth, M. and Groth, P.T. (2017). Storing, tracking, and querying provenance in linked data, IEEE Transactions on Knowledge and Data Engineering 29(8): 1751-1764.

Zakhary, V., Agrawa, D. and El Abbadi, A. (2017). Caching at the web scale, International Conference on World Wide Web Companion, Perth, Australia, pp. 909-912.

International Journal of Applied Mathematics and Computer Science

Journal of the University of Zielona Góra

Journal Information

IMPACT FACTOR 2017: 1.694
5-year IMPACT FACTOR: 1.712

CiteScore 2017: 2.20

SCImago Journal Rank (SJR) 2017: 0.729
Source Normalized Impact per Paper (SNIP) 2017: 1.604

Mathematical Citation Quotient (MCQ) 2017: 0.13

Cited By


All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 424 424 26
PDF Downloads 289 289 15