ATLAS-5381: Handle empty/corrupted reference vertices during entity delete and purge traversal - #740
Open
sheetalshah1007 wants to merge 2 commits into
Open
ATLAS-5381: Handle empty/corrupted reference vertices during entity delete and purge traversal#740sheetalshah1007 wants to merge 2 commits into
sheetalshah1007 wants to merge 2 commits into
Conversation
…elete and purge traversal
…dlerV1Test to cover addUpstreamProcessEntities, getColumnLineageEntities, and deleteEdgeReference
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes delete and purge failures when an owned or related reference vertex is empty or corrupt. The vertex and relationship edge remain, but required properties such as
__typeNameand__guidare missing.Error fixed:
ATLAS-400-00-014: Type ENTITY with name null does not existThis graph state is already handled on the read path, where broken relationship references can appear as
nullin arrays such asrelationshipAttributes.columns(ATLAS-4605). Before this fix, the same state could cause DELETE or PURGE to fail.This PR makes delete traversal resilient by skipping unreadable reference vertices and logging a warning. Valid vertices continue to be processed normally.
Changes in
DeleteHandlerV1isUnreadableEntityVertex()— newAdds a common check for vertices that cannot be safely read as Atlas entities.
A vertex is considered unreadable when it is:
nullor already removed__guid__typeNameThis check is used across the delete traversal paths to avoid passing invalid vertices to entity-header or type-resolution logic.
resolveEntityType()— newSafely resolves the entity type after validating the vertex.
If the vertex is unreadable or its type is unknown, the method returns
nullinstead of allowing type lookup to fail. This protects callers that need the entity type for dataset/process-specific delete handling.getOwnedVertices()— modifiedThis is the primary failure path for the original issue.
When traversing owned references such as a table's
columns, unreadable or unknown-type vertices are now skipped with a warning instead of throwingATLAS-400-00-014.Valid owned vertices and the parent entity continue through the normal delete flow.
accumulateDeletionCandidates()— modifiedUses
resolveEntityType()when determining the type of the root vertex.If the root vertex cannot be resolved, dataset/process-specific processing is skipped rather than failing with an exception. The vertex can still be included in deletion candidates so cleanup can proceed.
addUpstreamProcessEntities()— modifiedWhen deleting a dataset, Atlas can traverse linked process vertices.
Unreadable process vertices are now skipped before further processing, allowing the dataset delete to continue.
getColumnLineageEntities()— modifiedWhen deleting a process, Atlas can traverse column-lineage relationships.
Unreadable process or lineage endpoint vertices are now skipped, preventing a corrupt lineage reference from blocking the delete.
deleteEdgeReference()— modifiedWhen removing an edge, Atlas may update the entity at the other endpoint.
If that endpoint is unreadable, the inverse entity update is skipped. The edge deletion itself can continue without failing.
Tests
Four regression tests were added to cover the affected delete paths:
testDeleteTableWithHollowOwnedColumnVertexgetOwnedVertices()and owned-edge cleanuptestDeleteOutputTableWhenProcessVertexIsHollowaddUpstreamProcessEntities()testDeleteProcessSkipsHollowColumnLineageVertexgetColumnLineageEntities()testDeleteSubordinateWhenManagerVertexIsHollowdeleteEdgeReference()inverse updateEach test covers a separate scenario and removes the properties from the reference vertex to reproduce the corrupted graph state.
Existing
ATLAS-4766andATLAS-5317purge/resilience tests continue to provide coverage for related delete and purge scenarios.Expected Outcome
ATLAS-400-00-014nullin relationship arraysKnown Limitations
GETmay continue to shownullrelationship entries until the affected entity/edges are cleaned up.