[ZEPPELIN-6394] Make the interpreter setting write safe under concurrent saves

[dev-donghwan]

What is this PR for?

When several interpreters finish downloading their dependencies at the same time, each of them saves the interpreter settings from its own thread, so conf/interpreter.json is written concurrently. The reporter's log shows those writes failing:

ERROR ({Thread-16}) InterpreterSetting.java[run]:1003 - Fail to save interpreter.json
java.nio.file.NoSuchFileException: /data/soft/zeppelin-0.11.2/conf/interpreter.json
    at java.nio.file.Files.move(Files.java:1395)
    at org.apache.zeppelin.util.FileUtils.atomicWriteToFile(FileUtils.java:60)
    at org.apache.zeppelin.storage.LocalConfigStorage.save(LocalConfigStorage.java:54)
    at org.apache.zeppelin.interpreter.InterpreterSettingManager.saveToFile(InterpreterSettingManager.java:350)
    at org.apache.zeppelin.interpreter.InterpreterSetting$1.run(InterpreterSetting.java:1001)

There are two problems behind it.

1. The write itself is not atomic. FileUtils.atomicWriteToFile writes a temp file and then moves it over the destination with REPLACE_EXISTING alone. That replace is not a single operation, so when two writers move their own temp file onto the same destination, one of them can fail with NoSuchFileException and leave its temp file behind in the config directory. The failure is swallowed as an ERROR log, so the settings silently do not reach disk.

Measured on the current code with 16 threads writing the same file 300 times each:

move options	failed writes (of 4800)	leftover temp files
REPLACE_EXISTING	9 – 24, reproduced on every run	same count
REPLACE_EXISTING, ATOMIC_MOVE	0	0

This PR requests ATOMIC_MOVE as well. The temp file is created in the destination directory (Files.createTempFile(destinationDirectory, ...)), so the move never crosses file systems and the atomic move is always supported. The option was present as a trailing comment from the first version of this method in ZEPPELIN-4907 and was never enabled; there is no recorded reason for leaving it off.

2. The snapshot and the write are two steps. InterpreterSettingManager.saveToFile copies the settings and then writes them, without holding anything in between, so an older snapshot can be written after a newer one. When that happens, a setting that was added in between is dropped from the file while it survives in memory, and the loss only shows up after the next restart. This is the serialization the Jira asks for; it is done with a private lock rather than synchronized on the method so that the monitor stays internal.

Note for the Jira description: on the default LocalConfigStorage the concurrent writes do not corrupt the file, because every write goes to its own temp file. What they do is fail, and lose an update.

Relation to #5144. The saveToFile() serialization in this PR overlaps #5144, which was opened by the reporter and already carries an LGTM that a later rebase dismissed. That PR does not cover the move itself, which is what produces the NoSuchFileException in the report. If it is preferred to land #5144 for the serialization, I am happy to drop the InterpreterSettingManager change from here and leave only the FileUtils fix.

What type of PR is it?

Bug Fix

What is the Jira issue?

• https://issues.apache.org/jira/browse/ZEPPELIN-6394

How should this be tested?

Two new tests, both verified to fail without the corresponding change:

• FileUtilsTest.testConcurrentWritesToTheSameFileSucceed — 16 threads write the same file 300 times each and no write may fail or leave a temp file behind. Fails on master on every run with the NoSuchFileException above.
• InterpreterSettingManagerSaveRaceTest.testSaveDuringAnotherSaveIsNotOverwritten — a setting is added while another save is in flight, and the last write has to carry it. Fails on master with the last write dropped added_during_save.

Existing suites run: InterpreterSettingManagerTest, InterpreterSettingTest, InterpreterFactoryTest, InterpreterInfoSavingTest, LocalConfigStorageTest, CredentialsTest, LocalRecoveryStorageTest.

Questions:

• Does the license files need to update? No
• Is there breaking changes for older versions? No
• Does this needs documentation? No