Historical distributed-data systems laboratory.
LegacyBigDataLab is a 2026 sanitized public reconstruction of historical distributed-data laboratory work preserved from August 2023. The surviving material shows hands-on work around a Hortonworks HDP 2.6.5 sandbox, Docker-based lab setup, Oracle-oriented ingestion, Sqoop, Hive, Impala, Kerberos authentication steps, and Hadoop MapReduce tuning.
The original material also contains environment-specific credentials, internal network identifiers, operational metadata, logs, and third-party binaries. Those artifacts are not redistributed here.
This repository therefore preserves the architecture, parameter shapes, workflow behavior, and engineering decisions through documentation, reconstructed HiveQL, deterministic synthetic fixtures, and static validation tooling.
The preserved source bundle contains an adapted Docker sandbox plus custom ingestion tooling. File timestamps and runtime logs place the available evidence in August 2023. There is no .git directory in the supplied historical archive, so file-level authorship and commit chronology cannot be reconstructed safely.
The Docker base traces to the public vhbfernandes/hdp-sandbox project, which used the Hortonworks Sandbox HDP 2.6.5 image and was distributed under the MIT License. The local historical material added a custom Dockerfile, environment mounts, Hive database initialization, ingestion scripts, parameter files, logs, and operational notes.
See docs/provenance.md for the publication boundary.
- Preserved evidence period: August 2023.
- Direct runtime evidence: logs dated 2023-08-24.
- Public reconstruction: 2026.
The archive timestamps are secondary evidence; they are not a substitute for missing Git history.
Historical configuration only + 2026 synthetic/static validation.
The exact HDP runtime is not presented as a current supported platform and is not required to validate this repository. The public reconstruction intentionally avoids pretending that an obsolete legacy distribution is a modern reproducible deployment.
- adapting a containerized legacy Hadoop distribution for a local laboratory;
- parameter-driven Oracle-to-data-lake ingestion with Sqoop;
- dimension and fact workflow separation;
- date-partitioned fact ingestion;
- Hive staging and insert-overwrite patterns;
- configurable MapReduce split sizing and memory settings;
- Impala used to query ingestion-control metadata;
- Kerberos initialization before protected ingestion flows;
- explicit handling of execution priorities and custom date ranges;
- the operational reality that partial successes and downstream failures must be recorded rather than hidden.
The architecture separates the historical Docker/Compose runtime from the data-ingestion workflow. Docker Compose hosted the Hortonworks HDP 2.6.5 sandbox and its access proxy; Oracle remained an external source. The proxy is an access layer, not a data-processing stage.
For the ingestion sequence itself:
The public repository records the Docker topology as a sanitized, non-runnable evidence model under docker/. It intentionally does not ship the original Compose/Dockerfile runtime as a supported 2026 deployment.
The historical source does not preserve enough configuration to publish claims about cluster size, HDFS replication, YARN queues, fault tolerance, throughput, or performance gains.
| Component | Evidence status | Basis |
|---|---|---|
| Hortonworks Sandbox HDP 2.6.5 | Verified | Historical Dockerfile/image reference |
| Docker / Docker Compose | Verified | Local sandbox setup |
| Ambari | Verified | Historical README and exposed service configuration |
| Hive | Verified | HQL, CLI invocations, database initialization |
| Impala | Verified | Shell usage and operational documentation |
| Sqoop | Verified | Ingestion scripts and runtime logs |
| Oracle JDBC | Verified | Connection construction and JDBC driver artifacts |
| Kerberos | Verified | kinit/keytab workflow references |
| Hadoop MapReduce tuning | Verified | Hive settings for mapper/reducer memory and input splits |
| HDFS | Inferred, not directly configured | Sqoop target-directory semantics and HDP context; no preserved HDFS config |
| YARN | Unsupported as a project claim | No project-specific queue or ResourceManager configuration preserved |
Docker and Docker Compose are direct historical evidence, not merely contextual technologies. The preserved source contains a Compose 3.8 topology with a locally built sandbox-hdp service based on Hortonworks Sandbox HDP 2.6.5 and a separate sandbox-proxy access service.
The public reconstruction keeps this boundary explicit without pretending that the legacy runtime is currently supported or safely reproducible. See docker/README.md and docker/historical-topology.yml.
data/synthetic/ contains deterministic fixtures for a generic telecommunications-style analytical workflow. They reproduce representative schemas and workflow behavior, not original records, names, identifiers, infrastructure, or table contents.
Generate them with:
python3 scripts/generate_synthetic_data.pyValidate the reconstruction with:
python3 scripts/validate_reconstruction.pyRender a representative dry-run ingestion plan with:
python3 scripts/render_ingestion_plan.pyThe plan is documentation-oriented. It does not connect to Oracle, Hadoop, Hive, Impala, or any external environment.
- Read a parameter row describing a source and destination.
- Construct a Sqoop-style import plan.
- Land source rows in a distributed target directory.
- Use a staging table.
- Insert-overwrite the trusted dimension table.
- Select a date partition.
- Read mapper/split configuration from ingestion metadata.
- Construct a partition-filtered source query.
- Land the result.
- Insert-overwrite a Hive partition.
- Optionally tune input split minimum/maximum values.
The historical logs show successful Sqoop messages followed by Hive insert failures on 2023-08-24. This repository therefore does not claim a preserved end-to-end successful run.
The exact legacy runtime is intentionally not required. The 2026 validation harness checks:
- deterministic synthetic-data generation;
- parameter schema consistency;
- HiveQL reconstruction structure;
- representative ingestion-plan generation;
- repository publication boundaries;
- absence of historical private artifacts.
This public repository is a 2026 sanitized reconstruction of historical distributed-data laboratory work.
The original material included environment-specific configuration, internal infrastructure references, credentials, operational metadata, and binaries that are not redistributed.
Synthetic fixtures preserve representative schemas and workflows without exposing original records or credentials.
The reconstructed documentation, fixtures, validation scripts, and diagram are 2026 publication artifacts and must not be attributed to the historical period.
Excluded from this repository:
- credential and user files;
- keytab-related material;
- internal hostnames and private IP addresses;
- real operational logs and parameter rows;
- organization-specific table and field inventories;
- Oracle JDBC binaries and custom JARs;
- the historical
.bash_history; - raw third-party sandbox files that are not needed for this reconstruction.
See docs/security-boundary.md.
No real dataset is redistributed. The surviving historical archive primarily contains ingestion metadata, operational logs, connection/configuration material, and parameter extracts rather than a standalone redistributable source dataset.
The public fixtures are generated from code with a fixed seed and use fictional identifiers only.
- No historical Git metadata is present in the supplied archive.
- File-level authorship for custom historical artifacts is unresolved.
- The exact cluster topology is not preserved.
- HDFS is implied by the HDP/Sqoop workflow but no project-specific HDFS configuration is preserved.
- YARN configuration is not preserved.
- No verified performance or scale measurements are available.
- Historical logs preserve partial runtime evidence, including downstream Hive failures.
- Employer/team ownership of custom historical scripts is not established; those scripts are therefore not redistributed.
This project documents a historical step between relational/ETL work and later cloud data-platform engineering: hands-on exposure to distributed ingestion, metadata-driven orchestration, Hive transformations, and the operational constraints of a legacy Hadoop ecosystem.
The claim is deliberately narrow: historical evidence of distributed-data systems practice before the transition toward cloud data architecture.
No repository-wide open-source license is asserted for this reconstruction yet. The historical source mixes a third-party MIT-licensed sandbox base with custom material whose ownership is not fully preserved in the available evidence.
The public repository does not redistribute the third-party sandbox source or the historical custom scripts. See docs/provenance.md for attribution and rights handling.