BDEv Features
BDEv supports the following features, which are executed in an automatic way.
- BDEv performs:
- Automatic configuration of the frameworks
- Deployment of the frameworks using different cluster sizes
- Execution of predefined and user-defined workloads (batch and interactive modes)
- BDEv automatically records:
- Output and runtime of the workloads
- Performance and scalability results
- Configuration and log directories of the frameworks
- Resource utilization stats using dstat/dool and BDWatchdog
- CPU power consumption stats using RAPL and turbostat
- Microachitecture-level events using Oprofile
- Energy consumption stats using HPE iLO technology
- Automatically-generated graphs
- BDEv allows:
- To execute the workloads using different frameworks and cluster sizes
- To unify the configuration of the different frameworks
- To configure timeouts for the workloads
- To setup multiple disks to store intermediate data for the frameworks
- To configure different storage backends (HDFS, NFS)
- To leverage high-performance resources like IP over InfiniBand (IPoIB)
Supported frameworks in BDEv 4.0.0
| Framework | | Versions | | Deploy mode | | Network interfaces |
| Hadoop-YARN | | 2.10.2 / 3.0.3 / 3.1.4 / 3.2.4 / 3.3.6 / 3.4.3 / 3.5.0 | | YARN | | Ethernet / IPoIB |
| RDMA-Hadoop-3 | | 0.9.1 | | YARN | | Ethernet / IPoIB / RDMA (IB & RoCE) |
| Spark | | 2.4.8 / 3.0.3 / 3.1.3 / 3.2.4 / 3.3.4 / 3.5.9 / 4.0.4 / 4.1.3 / 4.2.0 | | Standalone / YARN | | Ethernet / IPoIB |
| Flink | | 1.15.4 / 1.16.3 / 1.17.2 / 1.18.1 / 1.19.3 / 1.20.5 | | Standalone / YARN | | Ethernet / IPoIB |
Supported benchmarks in BDEv 4.0.0
| Benchmark | | Input generator | | Hadoop | Spark | Flink |
| Testdfsio | | - | | ✔ | | |
| Wordcount | | RGen (Hadoop RandomTextWriter) | | ✔ | ✔ | ✔ |
| Sort | | RGen (Hadoop RandomTextWriter) | | ✔ | ✔ | ✔ |
| Grep | | RGen (Hadoop RandomTextWriter) | | ✔ | ✔ | ✔ |
| Terasort | | RGen (Hadoop TeraGen) | | ✔ | ✔ | ✔ |
| TPCx-HS | | RGen (TPC HSGen) | | ✔ | ✔ | ✔ |
| Pagerank | | RGen | | ✔ | ✔ | ✔ |
| Connected Components | | RGen | | ✔ | ✔ | ✔ |
| Bayes | | RGen | | ✔ | ✔ | |
| Kmeans | | RGen (Mahout GenKMeansDataset) | | ✔ | ✔ | ✔ |
| Aggregation | | RGen | | ✔ | ✔ | ✔ |
| Join | | RGen | | ✔ | ✔ | ✔ |
| Scan | | RGen | | ✔ | ✔ | ✔ |
