BDEv: Big Data Evaluator

News

[03/09/2026] BDEv 4.0.0 is released!

BDEv 4.0.0 is a is a major release that modernizes the orchestration stack, introducing support for Spark 4.x, Hadoop 3.5, and Flink 1.20 LTS, adding new benchmarks for Spark and Flink, while removing support for legacy framework versions. A key highlight of this release is the refactoring of the storage layer. BDEv v4.0.0 allows users to configure NFS as a lightweight storage backend alternative to HDFS for workloads that do not require distributed storage. Future minor releases will build upon this foundation to introduce support for additional distributed storage backends, such as Apache Ozone.

  • Updated versions of the following frameworks
    • Hadoop 3.4.3
    • Spark 3.4.4
    • Spark 3.5.9
    • Flink 1.19.3
  • Remove support of the following frameworks
    • Hadoop 1.x
    • Hadoop-UDA 1.x
    • Hadoop-UDA YARN
    • Spark 1.x
    • Flink < 1.15
    • RDMA-Hadoop 1.x
    • RDMA-Hadoop 2.x
    • RDMA-Spark
    • DataMPI
    • FlameMR
  • Support for the Spark 4.x line
  • Support for the Hadoop 3.5.x line
  • Support for the Flink 1.20.x line (LTS)
  • Support for NFS as a storage backend alternative to HDFS, which can be configured in bdev-conf.sh
  • Flink on YARN relies on the Session Mode instead of the deprecated Per-job Mode
  • Add Flink SQL benchmarks implementations (scan, join and aggregation) using the Table API
  • Update Flink implementations for PageRank and ConnectedComponents to avoid using the deprecated Graph API (Gelly)
  • Max value for Flink network memory size now also supports an auto-calculated value
  • Add Flink parameter to set the size of memory buffers used by the network stack and the memory manager (frameworks-conf.sh)
  • Add Spark benchmarks implementations using the Dataset API
  • Add Spark parameter to configure the number of shuffle partitions based on Executor cores (frameworks-conf.sh)
  • Add Spark parameter to configure Kryo registration behavior (frameworks-conf.sh)
  • Add Spark parameter to configure compression codec for Parquet files (frameworks-conf.sh)
  • Spark AQE is now enabled by default
  • Fix Spark memory configuration for Driver and AppMaster on YARN
  • Improve and simplify configuration for RDMA-Hadoop-3
  • RDMA interface for RDMA-Hadoop-3 can now be configured for IB and RoCE networks (system-conf.sh)
  • Add RDMA-Hadoop-3 parameter to configure the packet size for RDMA write (frameworks-conf.sh)
  • Support for starting the Secondary NameNode for HDFS
  • Allow to set the number of cores per mapper/reducer (mapreduce.sh)
  • Add cleanup phase for YARN applications and improve data cleanup
  • Add disk space sanity checks with configurable warning threshold (bdev-conf.sh)
  • HDFS and YARN services are now stopped using the Hadoop scripts
  • Improve Hadoop deployment by waiting for HDFS safe mode readiness
  • Update RGen data generator to support local file system URI (file://) and to include bug fixes
  • Update resource monitoring tool to dool v1.3.8
  • Update iLO scripts to version 6.00
  • Update RAPL-based power monitoring tool to PAPI v7.2.0
  • BDEv now runs by default in command mode when no benchmark is selected in benchmarks.lst
  • Replace legacy expect with native GNU timeout in benchmark runner
  • Improve command mode with configurable interactive shell (benchmarks-conf.sh)
  • Fix failure detection when running benchmarks
  • Simplified tool configuration and removed experiment directory
  • Fix Java detection and improve JAVA_HOME configuration
  • Multiple refactorings and code modernizations, minor changes and bug fixes

[11/04/2024] BDEv 3.9 is released!

BDEv 3.9 is a point release for the BDEv 3.x line

  • Updated versions of the following frameworks
    • Hadoop 3.3.6
    • Spark 3.2.4
    • Spark 3.3.4
    • Flink 1.14.6
    • Flink 1.15.4
  • Support for the Hadoop 3.4.x line
  • Support for the Spark 3.4.x and 3.5.x lines
  • Support for the Flink 1.16.x, 1.17.x, 1.18.x and 1.19.x lines
  • Add HDFS parameter to configure DataNode heartbeat (hdfs-default.sh)
  • Add YARN parameter to configure the scheduler class (yarn-default.sh)
  • Add several options to customize the YARN fair scheduler (yarn-default.sh)
  • Add YARN parameter to configure NodeManager heartbeat (yarn-default.sh)
  • Configuration of each experiment is now copied into the output report directory
  • HDFS formatting and data deletion can be configured (hdfs-default.sh)
  • HDFS replication factor is automatically set when there are not enough DataNodes
  • HDFS input data is reused if already exist
  • Update RAPL-based power monitoring tool to PAPI v7.1.0
  • Upate RGen data generator to include minor improvements and bug fixes
  • Update resource monitoring tool to dool v1.3.1
  • SSH optional parameters can now be set (bdev-default.sh)
  • Fix bug when running Spark and Flink on YARN
  • Remove support for Flink versions 1.12.x and 1.13.x
  • Minor changes and bug fixes

[29/07/2022] BDEv 3.8 is released!

BDEv 3.8 is a point release for the BDEv 3.x line

  • Updated versions of the following frameworks
    • Hadoop 2.10.2
    • Hadoop 3.2.4
    • Hadoop 3.3.3
    • Spark 3.2.2
    • Flink 1.14.5
  • Support for the Spark 3.3.x line
  • Support for the Flink 1.15.x line
  • Add new sorting benchmark for Hadoop, Spark and Flink: TPCx-HS
  • Update RGen data generator to support TPCx-HS (HSGen) and fix TeraGen bug with Hadoop 3.X
  • Hadoop classpath is now automatically configured for Spark and Flink
  • Add option to use hostnames or IPs for cluster nodes (bdev-default.sh)
  • Add Hadoop parameter for setting timeout on waiting response from server (core-default.sh)
  • Add HDFS parameter to configure write packet size (hdfs-default.sh)
  • Add HDFS parameter to configure retries when writing blocks to DataNodes (hdfs-default.sh)
  • Add HDFS parameter to configure handler count for DataNodes (hdfs-default.sh)
  • Add option to set an IP for loopback interface (system-default.sh)
  • Improve frameworks cleanup
  • Add sanity checks for networking setup
  • Remove support for Flink versions 1.11.x
  • Minor changes and bug fixes

[18/03/2022] BDEv 3.7 is released!

BDEv 3.7 is a point release for the BDEv 3.x line

  • Updated versions of the following frameworks
    • Hadoop 3.3.2
    • Spark 3.1.3
    • Flink 1.11.6
    • Flink 1.12.7
    • Flink 1.13.6
  • Support for the Spark 3.2.x line
  • Support for the Flink 1.14.x line
  • Add Flink configuration parameters for several shuffle settings
  • Remove support for Flink versions 1.10.x
  • Add sanity check for Flink configuration file
  • Update resource monitoring tool to dool v1.0.0
  • Minor changes and bug fixes

[16/07/2021] BDEv 3.6 is released!

BDEv 3.6 is a point release for the BDEv 3.x line

  • Updated versions of the following frameworks
    • Hadoop 2.10.1
    • Hadoop 3.1.4
    • Hadoop 3.2.2
    • Hadoop 3.3.1
    • Spark 2.4.8
    • Spark 3.0.3
    • Flink 1.10.3
    • Flink 1.11.3
  • Support for the Spark 3.1.x line
  • Support for the Flink 1.12.x and 1.13.x lines
  • Add Spark configuration parameters for
    • spark.memory.fraction
    • spark.memory.storageFraction
    • spark.kryoserializer.buffer.max
    • spark.executor.heartbeatInterval
  • Adaptative Query Execution (AQE) can be enabled and configured for Spark 3.x
  • Add Flink configuration parameters for several network memory settings and timeouts
  • Add HDFS configuration parameters for setting socket-related timeouts
  • Command batch mode now supports running multiple scripts stored in a directory
  • Remove support for Flink versions <= 1.9
  • All input datasets are now created using an internal data generator tool (RGen)
  • Set Hadoop classpath properly
  • Add sanity checks for some dependencies
  • Fix missing dool files
  • Minor changes and bug fixes

[07/09/2020] BDEv 3.5 is released!

BDEv 3.5 is a point release for the BDEv 3.x line

  • Updated versions of the following frameworks
    • Spark 2.4.6
    • Flink 1.9.3
    • Flink 1.10.2
  • Support for the Hadoop 3.3.x line
  • Support for the Spark 3.0.x line
  • Support for the Flink 1.11.x line
  • Update resource monitoring tool to dool v0.9.9, a Python 3 compatible clone of dstat
  • Update RAPL-based power monitoring tool to use PAPI v6.0.0
  • Update iLO scripts to version 5.30
  • RAPL-based monitoring tool now can report readings for uncore devices if PP0 is available
  • Fix bug when setting iLO credentials
  • Minor bug fixes

[25/03/2020] BDEv 3.4 is released!

BDEv 3.4 is a point release for the BDEv 3.x line

  • Updated versions of the following frameworks
    • Spark 2.4.5
    • Flink 1.8.3
    • Flink 1.9.2
  • Support for the Flink 1.10.x line
  • Allow to start the job history server for Flink
  • Simplified and improved cluster deployments for Spark & Flink
  • Add support to configure the YARN Timeline server (yarn-default.sh)
  • Add support to configure the disk health checker service for YARN (yarn-default.sh)
  • Add support to configure last access time precision for HDFS (hdfs-default.sh)
  • Add support to enable short-circuit local reads for HDFS (hdfs-default.sh)
  • Memory settings for some daemons (eg, DataNode) can now be set individually
  • Fix SQL and ConnComp workloads for Spark
  • Spark & Flink benchmarking suites are now downloaded on demand (removed from git repo)
  • Scala version for Spark & Flink can be set in solutions-default.sh (2.11 by default)
  • Set LD_LIBRARY_PATH in Spark & Flink to include the path to Hadoop native libraries
  • Hadoop existence is checked when deployed together with other frameworks
  • Minor bug fixes

[17/12/2019] BDEv 3.3 is released!

BDEv 3.3 is a point release for the BDEv 3.x line. From this version onwards, BDEv is hosted at GitHub

  • Updated versions of the following frameworks
    • Hadoop 2.10.0
    • Hadoop 3.1.3
    • Hadoop 3.2.1
    • Spark 2.3.4
    • Spark 2.4.4
    • Flink 1.8.2
  • Support for the Flink 1.9.x line (tested with Flink 1.9.1)
  • Add support for BDWatchdog tool, which allows per-process resource monitoring and JVM profiling
  • BDEv now runs by default in command mode when no framework is selected in solutions.lst
  • Update PAPI library to version 5.7.0
  • Update RAPL power monitoring tool to use PAPI 5.7.0
  • Add support to configure the binary names for Python2 and Python3 (system-default.sh)
  • Fix bug when running Flink in standalone mode
  • Other minor bug fixes

[06/06/2019] BDEv 3.2 is released!

BDEv 3.2 is a point release for the BDEv 3.x line

  • Updated versions of the following frameworks
    • Hadoop 2.7.7
    • Hadoop 2.8.5
    • Hadoop 2.9.2
    • Hadoop 3.1.2
    • Hadoop 3.2.0
    • RDMA-Hadoop-3 0.9.1
    • Flame-MR 1.2
    • Spark 2.3.3
    • Spark 2.4.3
    • Flink 1.7.2
    • Flink 1.8.0
  • Add support to configure handler count for Hadoop NameNode (hdfs-default.sh)
  • Add support to use unsafe based Kryo serializer for Spark (solutions-default.sh)
  • Fix bug when running multiple Executors per worker/node in Spark configuration
  • Minor bug fixes

[16/05/2018] BDEv 3.1 is released!

BDEv 3.1 is a point release for the BDEv 3.x line

  • Updated versions of some frameworks
    • Hadoop 2.7.6
    • Hadoop 2.8.3
    • Hadoop 2.9.0
    • Flame-MR 1.1
    • RDMA-Hadoop-2 1.3.5
    • Spark 2.3.0
    • RDMA-Spark 0.9.5
    • Flink 1.4.2
  • Support for the Hadoop 3.1.x line (tested with Hadoop 3.1.0)
  • Improved customization of experiments with setup and clean up commands
  • Extended configuration parameters

[07/11/2017] BDEv 3.0 is released!

This is a major release which contains a number of significant enhancements, along with new characteristics previously unreleased.

  • New monitoring tools
    • Power monitoring via RAPL
    • Microarchitecture-level event counting via Oprofile
  • Enhanced framework configuration parameters
  • Support for the Flink 1.3.x line (tested with Flink 1.3.2)
  • Support for the Spark 2.2.x line (tested with Spark 2.2.0)
  • Support for the Flame-MR 1.x line (tested with Flame-MR 1.0)
  • Updated versions of some frameworks
    • Hadoop 2.7.4
    • Hadoop 2.8.1
    • RDMA-Hadoop 1.2.0
    • RDMA-Spark 0.9.4
  • Updated version of resource monitoring tool to dstat 0.7.3
  • Improved graph generation portability

[25/11/2016] BDEv 2.3 is released!

BDEv 2.3 is a point release for the BDEv 2.x line

  • Added support for Flame-MR version 0.10.0
  • Added support for the Flink 1.1.x line (tested with version 1.1.3)
  • Added support for the SLURM job scheduler
  • Updated versions of some frameworks
    • Hadoop 2.6.5
    • Hadoop 2.7.3
    • RDMA-Hadoop-2 1.1.0
    • Spark 1.6.3
    • Flink 1.0.3
  • Memory settings for mappers and reducers can now be set separately
  • Added configuration variable in etc/core-default.sh to control the Hadoop parameter "io.file.buffer.size"
  • Virtual and physical memory limits in YARN containers as well as the ratio between them can now be configured in etc/yarn.default.sh
  • Fix critical bug when running Flink under YARN related with JobManager memory
  • Minor bug fixes

[04/07/2016] BDEv 2.2 is released!

BDEv 2.2 is a point release for the BDEv 2.x line

  • Add support for multiple Workers/TaskManagers per node in Spark and Flink
  • Enhancements in several Flink workloads
    • Grep
    • PageRank (with delta iterations)
    • Connected Components (with Gelly)
    • K-Means
  • Enhancements in several Spark workloads
    • TeraSort
    • K-Means (with MLlib)
  • Bug fixes

[05/05/2016] BDEv 2.1 is released!

BDEv 2.1 is a point release for the BDEv 2.x line

  • Allow to start the job history server for Spark-based frameworks
  • Added support for some new frameworks
    • Hadoop 2.6.4
    • RDMA-Hadoop-2 0.9.9
    • Spark 1.6.1
    • Flink 1.0.2
  • Added support for Mahout 0.11.2 and 0.12.0
  • Added new benchmarks for Spark
    • PageRank
    • Connected Components
    • K-Means
    • Bayes
    • Hive SQL queries:
      • Aggregation
      • Join
      • Scan
  • Added new benchmarks for Flink
    • PageRank
    • Connected Components
    • K-Means
  • Multiple bug fixes

[02/03/2016] BDEv 2.0 is released!

BDEv 2.0 is a major release which contains a number of significant enhancements. Starting with this release, the MapReduce Evaluator (MREv) tool has been renamed to Big Data Evaluator (BDEv).

  • Added support for Flink version 0.10.2 on YARN and standalone modes
  • Added support for RDMA-Spark version 0.9.1 on YARN and standalone modes
  • Added support for Spark on standalone mode
  • Updated versions of some frameworks
    • Hadoop-2.7.2
    • RDMA-Hadoop-2 0.9.8
    • Spark 1.5.2
    • Spark 1.6.0
  • New data set generation using the HiBench DataGen tool
  • Added new benchmarks for Hadoop-based frameworks
    • Grep
    • K-Means
    • Connected Components
    • Hive SQL queries:
      • Aggregation
      • Join
      • Scan
  • Added new benchmarks for Spark and Flink
    • WordCount
    • Sort
    • Grep
    • TeraSort
  • Added new benchmark for DataMPI
    • Grep
  • Allow to start the job history server for Hadoop-based frameworks
  • Input datasize for WordCount, Sort and TeraSort can now be different
  • Enhanced configuration of parameters
    • Separated configuration files for HDFS, MapReduce and YARN
    • Solution-specific parameters in a separated file
    • Many new configuration parameters
  • Support for multiple disks
  • Network interfaces are now optional parameters. If not specified, IPs are determined using the hostfile
  • Updated iLO scripts to version 4.70
  • Allow to configure IP, user name and password for iLO interfaces
  • Automatically download Apache Mahout and Apache Hive on demand
  • Optimized code refactorization and simplified internal configuration
  • Multiple bug fixes

[10/08/2015] MREv 1.1 is released!

MREv 1.1 has been released and can be obtained from the Downloads section. The main new features of MREv 1.1 are:

  • Updated versions of the frameworks
    • Hadoop-2.7.1-GbE
    • Hadoop-2.7.1-IPoIB
    • Hadoop-2.7.1-UDA
    • RDMA-Hadoop-2-0.9.7-GbE
    • RDMA-Hadoop-2-0.9.7-IPoIB
    • Spark-1.4.1-YARN-GbE
    • Spark-1.4.1-YARN-IPoIB
  • Configurable timeout for the workloads
  • User-defined batch command
  • Separate configuration and log directories for each evaluation
  • Enhanced resource configuration

[09/12/2014] MREv 1.0 is released!

Version 1.0 of MREv has been released and can be obtained from the Downloads section.